Getting data from a website?

Hi, currently I have some code using BeautifulSoup4 and TKinter to have a gui where it gets the HTML of a webpage I type in the GUI, but the problem is that the webpage I need has data loaded through javascript so BS4 is not getting that data. How would I be able to get that data? I've heard of selenium and PyQt4, but these both seem kind of difficult to install.

I need this program to work on a mac and windows, currently it works on a mac where I am developing it but would like to be able to move it to the windows computer when I am done and have an easy install. Please let me know what the best thing to do in my scenario is.

/r/Python
https://redd.it/8nnqj6

GUI Library help

Okay so I decided to make a browser as my next project but Ive heard that Tkinter isnt a good option and thats the GUI Library I am familiar with.. I was thinking something like KIVY I heard its good for a lot of things, but I have no idea how to do this at all to be honest

/r/Python
https://redd.it/8npgac

In Memory cacheing or not with Django

Hiya All

Been a long time lurker and would love and would love some info on people good at web technologies.

So use case. I'm lead developer on a BI team and for some quirk of history I'm rewriting a call center application for a global company hosted on AWS. Atm the app is a monolith of a frontend and DB backend, its legacy code built by someone for whom it was their first "programming project" so it's all spaghetti code and completely useless to try and refactor . but our operators are in Asia and America. I'm rebuilding the application to be a seperated backend and frontend, the backend will be written again in Django and atm I've built it as a REST API that can handle request for data. End users will use a front end to request sets of data to do with leads and property listings and then call up those leads.

The way im currently conceptualising this is we have our database backend and a REST API in django sitting in europe, then our end users working round the world make a request to it at the start of the session, get all of their leads and listings through the API then they're held in memory for the users session and they just post individual requests that we can deal with asynchronously \(celery \+ rabbitmq most likely\).

My question is, would it be better to go for an approach where an end user logs in, and receives all the listings and all the leads at the start of the session and holds them in memory then posts to the API with single entries, or is it better to make lots of individual requests to the backend based on user interaction \(I'd imagine i'd implement some sort of drop down menu in react\) which saves on a 5\-10 second load time \(just on my local machine, for someone in mexico requesting a server in europe load times would be longer\) . And if we are to hold this stuff in memory for an end user, what would be the normal approach for this?

/r/django
https://redd.it/8nko3j

Advice on making a better form submit button? (success_url)

With the django class based FormView, I'm not sure the simplest way to use the success_url.


I mean, I don't want the user to go to another "thank you" page.


I would like something else like for example, that the form vanishes and the thank you message takes it's place.


Or, a modal appears on top of it with the tank you message.


Or just a div gets displayed with a message beneath the form "your message is sent, thanks!".


Just some ideas, but I'm not sure how to accomplish this, I could most likely figure it out.. but I like learning shortcuts and I'm sure other people would like to know this as well. Good info to note down for later use, right?


I'm not using any SPA right now, I intent to learn to use react or vue later though but right now it's not in my toolset.


I know how to do everthing I said without Django btw.. it's with Django that I'm not sure how to go about this.

/r/djangolearning
https://redd.it/8nobv5

HELP - How to use application context inside the blueprint module model which reflects exsiting db

Hi,
I'm building a flask application. Here is the structure:

root_folder
|config.py
|manage.py
|app
|__init__.py
|mod_a
|mod_b
|__init__.py
|controller.py
|model.py
|view.py


in manage.py:

from app import create_app, db
from flask_script import Manager

app = create_app('production')
app.app_context().push()
manager = Manager(app)

@app.shell_context_processor
def make_shell_context():
return {
'db': db,
'User': app.auth.models_user.User,
}

if __name__ == '__main__':
manager.run()

in __init__.py:

bootstrap = Bootstrap()
db = SQLAlchemy()
migrate = Migrate()
bcrypt = Bcrypt()
authtoken = HTTPTokenAuth(scheme='Token')
api = Api()

def create_app(config_name):
app = Flask(__name__)
app.config.from_object(config[config_name])
/////regist all the blueprint
db.init_app(app)
migrate.init_app(app, db)
bootstrap.init_app(app)
bcrypt.init_app(app)
api.init_app(app)
return app


Now I started to write code in models.py in mod_b, it needs to reflect existing database, I'm doing this:

from app import db
from sqlalchemy import MetaData

meta_db1 = MetaData()
meta_db1.reflect(db.get_engine(bind='my_db1'))
meta_db2 = MetaData()
meta_db2.reflect(db.get_engine(bind='my_db2'))

class Table1(db.Model):
__table__ = meta_db2.tables['table1']
__bind_key__ = 'my_db2'

def __repr__(self):
return self.__name__

class Table2(db.Model):
__table__ = meta_db1.tables['table2']
__bind_key__ = 'my_db1'

def __repr__(self):
return self.__name__
It always complains that Working outside of application context. Tried to import app, but in app there is no app_context()
What should I do to make sure models & reflection will have the correct application context? Thanks a lot for your help

/r/flask
https://redd.it/8nqsra

How to Prefetch_related with Formsets?

I'm getting too many duplicated queries in a UpdateView, one from each Formset instances and I don't know how to apply *prefetch\_related* with this case.

I've three models: Entry, that has multiple instances of Boxes, and each one has a FK to Material. So, in one Entry the user has multiples Boxes and each one has one Material associated. The problem is, when the user edits the Entry, for every Box he has, Django hits the DB looking for the material. How can I implement *prefetch\_related* \(or maybe *select\_related*?\) to solve this?

This is my code:

**models.py**

class Entry(models.Model):
....
# No Foreign keys here

class Box(models.Model):
....
material = models.ForeignKey('Material', on_delete=models.CASCADE)

class Material(models.Model:
....
# No Foreign keys here

**views.py**

class EntryUpdateView(UpdateView):
model = Entry
form_class = EntryForm

def get_context_data(self, **kwargs):
data = super(EntryUpdateView, self).get_context_data(**kwargs)
data['materials'] = Material.objects.all()
return data

def get(self, request, *args, **kwargs):
self.object = self.get_object()
form_class = self.get_form_class()
form = self.get_form(form_class)
box_form = BoxUpdateFormSet(instance=self.object)

return self.render_to_response(self.get_context_data(form=form, box_form=box_form))

**models.py**

class EntryForm(ModelForm):
class Meta:
model = Entry
fields = ...

class BoxForm(ModelForm):
class Meta:
model = Box
fields = ....

BoxUpdateFormSet = inlineformset_factory(Entry, Box, form=BoxForm, can_delete=True, extra=0)

/r/django
https://redd.it/8nqldy

‘Russian Fake Tweets Visualized
https://towardsdatascience.com/russian-fake-tweets-visualized-6f73f767695

/r/Python
https://redd.it/8nqz6k

Decision Tree Algorithm With Example _ Using Python
https://www.youtube.com/watch?v=RmajweUFKvM

/r/Python
https://redd.it/8nrqmr

Visibility graph simulator built using Pyvisgraph

/r/Python
https://redd.it/8nsems

Developing a live-data fed dashboard. What is the best route to take?

Hello /r/python!

I am wanting to develop a dashboard that updates in real-time. (I'm playing around with RFID and want the numbers to adjust every time a card is scanned).

I've been researching Dashboards these past few days and did not realize how difficult it would be to decide which way to do this.


So if there are any Python devs here who has developed one of thse, which method did you take?
What I'm seeing now is:

* Vue.js + Django

* Flask

* Python script that feeds into a Django websocket

* Dash

* Plotly

* PyQt5

The most important aspect I need is data updating instantly. And obviously not anything incredibly difficult. I just want to see the data. Not make 110 awesome interactive graphs.

There are probably options I am forgetting. I love the Vue.js route but I am not really wanting to learn another language just yet as I don't feel I am where I want to be with Python.

Would anyone happen to have some advice?

Thanks!

/r/Python
https://redd.it/8nsf6k

[AF] Import issues with my project. Unsure how to approach.

I have a flask app that relies on web scraping to display data to the end user and let them download it. Right now, the various scrapers I have are sitting in the root level of the *myapp/* package/project so that they can write to app.db, a SQLite database.

Here is the current structure of the app:

my_app/
- app/
--- __init__.py
--- routes.py
--- models.py
--- templates/
--- static/
- scrapers/ (*note: my goal is create a folder like this)
--- scraper1
--- scraper2
--- scraper3
- config.py
- app.db
- scrape.py (*note: where my scraper currently is)


In the scrape.py file, I'm importing with the following import statement:

from app.models import db, table_name

My goal is to move scrape.py (and other scraping scripts) into a single directory where they can import **db** and **table_name** from app/models.py, but relative imports haven't worked for me.

Am I approaching this wrong?

/r/flask
https://redd.it/8nsv2a

Easier data analysis in Python with pandas (30+ videos)
http://www.dataschool.io/easier-data-analysis-with-pandas/

/r/Python
https://redd.it/8nxfwf

What's up with all the dict key->attribute frameworks (python-box, dataclasses, namedtuple etc.)?

### Data Classes are a-comin'

So, now that we're getting [data classes](https://www.python.org/dev/peps/pep-0557/), we've finally got a concise and convenient way to avoid huge `__init__` methods full of `self.foo = foo`. That's great, that's fantastic, that's overdue. I'm sick of writing an entire `__init__` method for things that have like four attributes.

But it made me think -- why are there so many frameworks for taking things that would ordinarily be dictionaries and making their 'keys' available as attributes?

### The alternatives
####Namedtuple

`Namedtuple` kind of has a reason to exist other than as a key->attribute interface. Even with dataclasses, it provides the only super-easy framework to represent a datatype or small object:

Employee = namedtuple('Employee', ['id', 'name', 'salary'])

I like Namedtuple. I think that the syntax is a little silly because of that mandatory first argument, but that's just adds character. Namedtuple instances are immutable, too, and that's useful in its own way.

####Box
Box (`python-box` on pypi) is something I only just used for the first time today, and it's basically the same kind of thing, but more magic:

>>> bands = Box({
"My Awesome Band": {
"Albums": ["Foo, Vol 1.", "Bar Baz Blues"],
"Artists": ["Mr. Example", "John Doe"]
}
})

>>> print(bands.My_Awesome_Band.artists[0])

Mr. Example

It's super awesome and I immediately started using it to keep track of a deep, multi-layer dictionary I got courtesy of some JSON API. Having ['many']['layers']['of']['these']['selectors'] can get annoying.

Even dataclasses can't beat it: a `dict` in a dataclass is still dataclass.mydict['and']['then']['you']['come']['back']['here']. A `dict` in a `Box` has its keys exposed too.

### A clear trend

This entire situation reminds me of type hints, where despite the fact that Python will officially always be dynamically typed, the Python maintainers eventually sighed and admitted maybe there was some use for type annotations in programming, even if they were just so much syntactic sugar for Python itself.

Isn't it pretty apparent by this point that people like to access attributes with dots? Accessing them by ['string keys'] is fun and all, but it's ugly-looking and repetitive. Dots are cleaner. A dot-access class might lose case-sensitivity or might require you to use no special characters in keys or whatever, but those are just quibbles.

Would the community support adding something more like Box to the standard library? Should there be some kind of `collections.DottedDict`? Should we just admit, yeah, dots are nice, and while they're not the official syntax for dictionaries, they're available if you like 'em?


/r/Python
https://redd.it/8nx4sk

Newbie ForeignKey list problem

Hello,

Again I'm struggling with something quite simple.

I have two models. Products and ParoductCategory

*class* ProductCategory\(*models**.Mode*l\):
category\_name = models.CharField\(*max\_length*=40\)
slug = models.SlugField\(*max\_length*=40,* uniqu*e=True\)
*class* Product\(*models**.Mode*l\):
product\_name = models.CharField\(*max\_length*=255\)
product\_amount = models.IntegerField\(\)
updated = models.DateTimeField\(*auto\_now*=True\)
timestamp = models.DateTimeField\(*auto\_now\_add*=True\)
category = models.ForeignKey\('ProductCategory', *on\_delete*=models.CASCADE,* related\_nam*e='cat'\)
additional\_info = models.TextField\(*blank*=True\)

I would like to iterate over ProductCategory and create list of items in each productcategory model. Help me guys plz, I won't sleep if i won't understand where is my problem.

I' trying to use CBV, and I was playing with get\_context\_data method in view.

/r/djangolearning
https://redd.it/8nwr6z

I'd like to create a convenience app that works across all apps--where do I begin?

Vague title, I know. Lemme explain:
I'm trying to create a text translator that works across not only the browser, but on the desktop or any other app--as long as the text is 'highliteable', I want to be able to hold a key down and hover over the text and have a little pop-up appear and translate the text.

So what exactly am I looking for? A desktop app using tkinter? How do I sense that the mouse is hovering over text and that it's highlighted? Pyautogui?
All I know so far is that it'll involve a translation API.
What other libraries will be involved?


Feedback much appreciated!

/r/Python
https://redd.it/8nyn8l

Django bugfix release: 2.0.6
https://www.djangoproject.com/weblog/2018/jun/01/bugfix-release/

/r/django
https://redd.it/8ntc2r

When is a project due for a complete re-write?

I built a project that has been in production for about 4 years now and is integral to the daily operations of the business. Its original design and functionality was for a fairly narrow set of processes. Over the years as the company has grown, I've tacked on features, built in new work arounds, and expanded the project beyond its initial intent. It's all working fairly well, but I'm definitely seeing how it's aging. Right now I'm working on adding a new feature that, were the project designed differently/better initially, I wouldn't even have to write this new code \-\- the project would already be able to handle the new requirement.

Over the last year or two, I've often wondered if there's a valid case for "we need to rebuild this beast". Management is reluctant \(understandably\) since their business rides on the project working in its current state.

Any of you have experience on this front?

/r/django
https://redd.it/8nvw8c

[N] UC Berkeley Open-Sources 100k Driving Video Database
https://medium.com/@Synced/uc-berkeley-open-sources-100k-driving-video-database-dce09ff7cf78

/r/MachineLearning
https://redd.it/8ns7vv

[AF]Why is my flask/SQLAlchemy simple app not respecting data constraints from db.Column parameters?

Hi.

I feel like this is more of SQLAlchemy question but I would assume that a lot of people here have used it as well in their Flask app, so this is why I'm asking here.


(Whole code at the end of the post)


The way I understand it, when creating the db using SQLAlchemy, we define the data that should be accepted by this column of the database using db.Column, for example:

field0 = db.Column("f0", db.String(3), unique=False, nullable=False, primary_key=True)


Here, the **field0** column is named *f0*, should accept only *string* data that is at most *3* characters long, should accept *non-unique* entries, but should not accept *empty* inputs.


However, once I run my app, I can enter literally anything into the respective columns, regardless of the constrains imposed during creation of the db.


So, either I did something wrong, or I don't understand how this whole thing should work.


I do understand that the db is defined the moment we create it, and not the moment the app is started. I made sure to create a fresh db with the correct column definitions.


**the_app.py**

____________________________

import os, datetime
from flask import Flask, Markup, render_template, request
from flask_sqlalchemy import SQLAlchemy

project_dir = os.path.dirname(os.path.abspath(__file__))
app = Flask(__name__)

database_file_uri = "sqlite:///{}".format(os.path.join(project_dir, "databse.db"))
app.config["SQLALCHEMY_DATABASE_URI"] = database_file_uri
db = SQLAlchemy(app)

class MyTable(db.Model):
field0 = db.Column("f0", db.String(3), unique=False, nullable=False, primary_key=True)
field1 = db.Column("f1", db.Integer, unique=False, nullable=False, primary_key=True)

def __repr__(self):
return " string representation".format(self.title)

def __init__(self, field0=field0, field1=field1):
self.field0 = field0
self.field1 = field1

@app.route("/", methods=["GET", "POST"])
def home():
if request.form:
newRow = MyTable(field0 = request.form.get("f0"), field1 = request.form.get("f1"))
db.session.add(newRow)
db.session.commit()

data_from_db = MyTable.query.all()
return render_template("home.html", data_from_db=data_from_db)

if __name__ == '__main__':
app.run(host='0.0.0.0', port=8080)


**templates\home.html**
____________________________

<html>
<body>
<h1>Add entry</h1>
<form method="POST" action="/">
<input type="text" name="f0">
<input type="text" name="f1">
<input type="submit" value="Add">
</form>
<h1>Values</h1>
{% for elem in data_from_db %}
<p>{{elem.field0}} - {{elem.field1}}</p>
{% endfor %}
</body>
</html>


**db initialization**
____________________________

from the_app import db
db.create_all()
exit()

To recap, column constraints:

field0 = db.Column("f0", db.String(3), unique=False, nullable=False, primary_key=True)
field1 = db.Column("f1", db.Integer, unique=False, nullable=False, primary_key=True)

But, when I run the app and go to http://localhost:8080/, I am:

- able to insert very long strings into field0 (greater in length than 3)

- able to insert strings into field1 (which should only accept integers)

- able to insert empty fields (despite nullable=False)

- unable to insert identical fields (despite unique=False)

Can someone explain why db seems to just completely ignore the column constrains I'm using? Thanks for any replies in advance.

/r/flask
https://redd.it/8o0mfd

Flask SQLAlchemy tenant data isolation

hello,

I'm currently moving my flask application from a multi instance to multi-tenant architecture. I want to isolate customer data to its own db rather than comingle it with context in a single db. I have found a couple of learning resources so far, like this bit on using binds for table isolation

SQLALCHEMY_DATABASE_URI = 'postgres://localhost/main'
SQLALCHEMY_BINDS = {
'users': 'mysqldb://localhost/users',
'appmeta': 'sqlite:////path/to/appmeta.db'
}
[Source](http://flask-sqlalchemy.pocoo.org/2.3/binds/#binds)

My trouble is whether this is the best way of isolation. I'm wondering how to update the binds and inject the new database URI as the application gains new tenants. The best resource I've found is
[this talk](https://www.infoq.com/presentations/saas-python).

I'm wondering if someone can help point me towards learning resources or help me ask this question in a better way because I'm stuck.

/r/flask
https://redd.it/8o15hq