How to query whole database with flask-sqlalchemy?

I have been working on a flask api to query data from database and return to user on each request, using flask\-sqlalchemy orm.

But problem is the database have over 200 tables and initializing model class for each table is something i don't want to do. I hope there is some alternative to do such task.

/r/flask
https://redd.it/8qs66b

DRF GenericAPI GET request taking 500ms to complete, generic.View GET request taking 10ms. What's happening?

I have a class that will check if a value is cached. If it is cached, then it will return the value from a local cache. Otherwise, this value will be generated by a remote database. If I use the DRF APIView, then the call will take around 500ms. If I use the standard View class in Django, then it takes about 10ms to return. I have checked, and both of these are hitting the cache. Is there any magic with the DRF APIView that could be responsible for this insane amount of overhead that I am getting?


class ItemDetail(views.APIView):

def get(self, request, pk):
key_name = 'item:{}'.format(pk)
data = cache.get(key_name)

# data should always already exist in the cache, but just in case
if not data:
obj = get_object_or_404(Item, pk=pk)
data = ItemSerializer(obj).data
# expiration set to None to preserve key
cache.set(key_name, data, None)

return JsonResponse(data)

/r/django
https://redd.it/8qvf4a

[D] Machine Learning - WAYR (What Are You Reading) - Week 44

This is a place to share machine learning research papers, journals, and articles that you're reading this week. If it relates to what you're researching, by all means elaborate and give us your insight, otherwise it could just be an interesting paper you've read.

Please try to provide some insight from your understanding and please don't post things which are present in wiki.

Preferably you should link the arxiv page (not the PDF, you can easily access the PDF from the summary page but not the other way around) or any other pertinent links.

Previous weeks :

|1-10|11-20|21-30|31-40|41-50|
|----|-----|-----|-----|-----|
|[Week 1](https://www.reddit.com/r/MachineLearning/comments/4qyjiq/machine_learning_wayr_what_are_you_reading_week_1/)|[Week 11](https://www.reddit.com/r/MachineLearning/comments/57xw56/discussion_machine_learning_wayr_what_are_you/)|[Week 21](https://www.reddit.com/r/MachineLearning/comments/60ildf/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 31](https://www.reddit.com/r/MachineLearning/comments/6s0k1u/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 41](https://www.reddit.com/r/MachineLearning/comments/7tn2ax/d_machine_learning_wayr_what_are_you_reading_week/)|||
|[Week 2](https://www.reddit.com/r/MachineLearning/comments/4s2xqm/machine_learning_wayr_what_are_you_reading_week_2/)|[Week 12](https://www.reddit.com/r/MachineLearning/comments/5acb1t/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 22](https://www.reddit.com/r/MachineLearning/comments/64jwde/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 32](https://www.reddit.com/r/MachineLearning/comments/72ab5y/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 42](https://www.reddit.com/r/MachineLearning/comments/7wvjfk/d_machine_learning_wayr_what_are_you_reading_week/)||
|[Week 3](https://www.reddit.com/r/MachineLearning/comments/4t7mqm/machine_learning_wayr_what_are_you_reading_week_3/)|[Week 13](https://www.reddit.com/r/MachineLearning/comments/5cwfb6/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 23](https://www.reddit.com/r/MachineLearning/comments/674331/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 33](https://www.reddit.com/r/MachineLearning/comments/75405d/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 43](https://www.reddit.com/r/MachineLearning/comments/807ex4/d_machine_learning_wayr_what_are_you_reading_week/)||
|[Week 4](https://www.reddit.com/r/MachineLearning/comments/4ub2kw/machine_learning_wayr_what_are_you_reading_week_4/)|[Week 14](https://www.reddit.com/r/MachineLearning/comments/5fc5mh/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 24](https://www.reddit.com/r/MachineLearning/comments/68hhhb/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 34](https://www.reddit.com/r/MachineLearning/comments/782js9/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 44](https://www.reddit.com/r/MachineLearning/comments/8aluhs/d_machine_learning_wayr_what_are_you_reading_week/)|
|[Week 5](https://www.reddit.com/r/MachineLearning/comments/4xomf7/machine_learning_wayr_what_are_you_reading_week_5/)|[Week 15](https://www.reddit.com/r/MachineLearning/comments/5hy4ur/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 25](https://www.reddit.com/r/MachineLearning/comments/69teiz/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 35](https://www.reddit.com/r/MachineLearning/comments/7b0av0/d_machine_learning_wayr_what_are_you_reading_week/)||
|[Week 6](https://www.reddit.com/r/MachineLearning/comments/4zcyvk/machine_learning_wayr_what_are_you_reading_week_6/)|[Week 16](https://www.reddit.com/r/MachineLearning/comments/5kd6vd/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 26](https://www.reddit.com/r/MachineLearning/comments/6d7nb1/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 36](https://www.reddit.com/r/MachineLearning/comments/7e3fx6/d_machine_learning_wayr_what_are_you_reading_week/)||
|[Week 7](https://www.reddit.com/r/MachineLearning/comments/52t

6mo/machine_learning_wayr_what_are_you_reading_week_7/)|[Week 17](https://www.reddit.com/r/MachineLearning/comments/5ob7dx/discussion_machine_learning_wayr_what_are_you/)|[Week 27](https://www.reddit.com/r/MachineLearning/comments/6gngwc/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 37](https://www.reddit.com/r/MachineLearning/comments/7hcc2c/d_machine_learning_wayr_what_are_you_reading_week/)||
|[Week 8](https://www.reddit.com/r/MachineLearning/comments/53heol/machine_learning_wayr_what_are_you_reading_week_8/)|[Week 18](https://www.reddit.com/r/MachineLearning/comments/5r14yd/discussion_machine_learning_wayr_what_are_you/)|[Week 28](https://www.reddit.com/r/MachineLearning/comments/6jgdva/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 38](https://www.reddit.com/r/MachineLearning/comments/7kgcqr/d_machine_learning_wayr_what_are_you_reading_week/)||
|[Week 9](https://www.reddit.com/r/MachineLearning/comments/54kvsu/machine_learning_wayr_what_are_you_reading_week_9/)|[Week 19](https://www.reddit.com/r/MachineLearning/comments/5tt9cz/discussion_machine_learning_wayr_what_are_you/)|[Week 29](https://www.reddit.com/r/MachineLearning/comments/6m9l1v/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 39](https://www.reddit.com/r/MachineLearning/comments/7nayri/d_machine_learning_wayr_what_are_you_reading_week/)||
|[Week 10](https://www.reddit.com/r/MachineLearning/comments/56s2oa/discussion_machine_learning_wayr_what_are_you/)|[Week 20](https://www.reddit.com/r/MachineLearning/comments/5wh2wb/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 30](https://www.reddit.com/r/MachineLearning/comments/6p3ha7/d_machine_learning_wayr_what_are_you_reading_week/)|[Week 40](https://www.reddit.com/r/MachineLearning/comments/7qel9p/d_machine_learning_wayr_what_are_you_reading_week/)||

Most upvoted papers two months ago:

/u/Molag_Balls: "Variance-based Gradient Compression for Efficient Distributed Deep Learning" A [proposed ICLR paper](https://openreview.net/forum?id=rkEfPeZRb) on a method for reducing the need for communication between workers in a distributed training environment.

/u/theainerd: AndrewNg released the first 19 chapters of his book Machine Learning Yearning. It is focused not on teaching you ML algorithms, but on how to make ML algorithms work. [Machine Learning Yearning](http://www.mlyearning.org/)

Besides that, there are no rules, have fun.

Note: The /u/ML_WAYR_bot was great, please bring it back :)

/r/MachineLearning
https://redd.it/8qp3at

Daemonizing django-rq workers in production

Hi all,

I'm a bit lost attempting to deploy an app that uses redis as a queue.

I'm deploying the app via Elastic Beanstalk and have automated the deployment in the .ebextensions folder (e.g. wget, make,
daemonize server, run etc) but am having trouble on how to figure out how to run workers.

Do I ssh in to my ec2 instance and type "python manage.py rqworker"?

Also should I use anything else to manage workers and the queue?

Thanks!

Edit: my redis.config file for elastic beanstalk looks like this:

sources:
/home/ec2-user: http://download.redis.io/releases/redis-4.0.10.tar.gz
commands:
redis_build:
command: make
cwd: /home/ec2-user/redis-4.0.10
redis_config_001:
command: sed -i -e "s/daemonize no/daemonize yes/" redis.conf
cwd: /home/ec2-user/redis-4.0.10
redis_config_002:
command: sed -i -e "s/# maxmemory <bytes>/maxmemory 500MB/" redis.conf
cwd: /home/ec2-user/redis-4.0.10
redis_config_003:
command: sed -i -e "s/# maxmemory-policy volatile-lru/maxmemory-policy allkeys-lru/" redis.conf
cwd: /home/ec2-user/redis-4.0.10
redis_server:
command: src/redis-server redis.conf
cwd: /home/ec2-user/redis-4.0.10


/r/djangolearning
https://redd.it/8qyslb

How to open matplotlib plots in a popup window?

Seen this being asked several times, but most seem outdated. Bash magic %matplotlib qt doesn't seem to work, the graphs still appear but inline.
I tried %matplotlib qt5 which had some missing dependencies on my Ubuntu machine.
Any suggestions?
Thanks!

/r/IPython
https://redd.it/8qvn0p

Data Sets and Challenge Statements Released for this year's Hack for the Sea

The [Hack for the Sea](https://hackforthesea.tech) Crew is proud to present this year's challenge statements and data sets.

They are, as follows:

* [How does a changing coastal watershed impact coastal waters?](https://hackforthesea.tech/GLO/challenge/1)
* [Can you predict where and when cod spawning will occur?](https://hackforthesea.tech/GLO/challenge/2)
* [Can you design a mooring that's both eelgrass and user-friendly?](https://hackforthesea.tech/GLO/challenge/3)
* [Can an individual whale be identified based on its blowhole?](https://hackforthesea.tech/GLO/challenge/4)

The event is all ages and open to anybody who is ready and willing to provide their skills to help the oceans. Also, while the summit will be held in person, the community is open and involved year round. Join us!

/r/pystats
https://redd.it/8qz3xb

Mechanical Engineer Trying to Create a Python Data Aquistion GUI

So a little of background, I'm a mechanical engineer that doesn't a lot of testing and hates NI/Labview. I've been teaching my self python via various books, so I would still very much call myself a beginner. I've recently purchased some data acquisition hardware and supplier has supplied an API and some example code for interfacing with the hardware but I would like to create a GUI so that it is more user friendly and other people can use the system with needing to know how to understand the python code. That being said I know there are a lot of frameworks out there to develop a GUI and I was wondering if anyone had any experience and/or recommendations. For reference the DAQ unit is from UEI.

/r/Python
https://redd.it/8qyjrh

What is the best way of doing integration test with flask?

I am trying to make an integration test using thread, using multiple ports. But it is not working very well. What is the best way of doing it?

def get_server():
Server = Thread(target=main)
Server.daemon = True
return Server

def test_update_blockchain():
sys.argv = ['','--port', '8002','--wallet','1']
ip1 = f'http://localhost:8002'
server1 = get_server()
server1.start()
time.sleep(0.01)

sys.argv = ['','--port', '8003', '--wallet', '2', '--servers', 'localhost:8002']
ip2 = f'http://localhost:8003'
server2 = get_server()
server2.start()
time.sleep(0.01)

requests.get(f'{ip1}/mine')
res = requests.get(f'{ip2}/blockchain')
assert res.status_code == 200
blockchain = res.json()
assert len(blockchain['blocks']) == 1

/r/flask
https://redd.it/8qz382

Is it a good idea to make a text-based story game in Python?

I'm doing this because Python is like the easiest language but I don't know if I should use it. Help?

/r/Python
https://redd.it/8qyhn6

Server-Sent Events for Django
https://github.com/fanout/django-eventstream

/r/django
https://redd.it/8qxjfw

how do you prevent this human error before deploy?

often I need to change some code in order to access a part of my app I wouldn't otherwise be able to reach (e.g. access a payment fail view by hand coding in success is always False). It is super easy to forget to correct such changes before commit or even push to server.

How do you catch these human errors? I guess an option is writing a unit test to explicitly make sure you undo the mods. Seems a bit overkill though.

I find myself wanting to be able to label my change (eg #hack) and then detect for any such labels on deploy to stop deploy. Again, perhaps overkill?


/r/django
https://redd.it/8r1gha

Renaming the main flask file

I have a flask app where the main .py file has a hyphen in the filename, and now that I’m trying to run some shell commands I realize that the hyphen is going to be giving me invalid character syntax errors.

I renamed that file to no longer have the hyphen, and then updated the path in two files in my app/py_cache file which referenced that old file name, but I’m now getting errors when I try to start my server, since it still appears to be looking for the old file name somewhere. I can’t tell where the error is coming from, and I’ve searched through all my app files but I can’t find any other references to the old name.

I’ve tried searching for info about this but somehow to no avail so far... Any ideas what I could be missing here? Or is there a better way to rename it than manually changing the paths in those files?

/r/flask
https://redd.it/8qwaz1

Kyso - Publish Your Jupyter Notebooks as Blog Posts

Hi Guys - the founders of kyso here.

Kyso (https://kyso.io) is a blogging platform for data science. We render your jupyter notebooks, including all code, interactive visualisations and rich markdown, as awesome data blogs.

Upload existing notebooks, import your repos from Github and start brand new Jupyterlab environments in the cloud, all for free.

And then publish and share your blogs with the community!

Have a look here for a few example posts on our platform:

https://kyso.io/jamesle/fifa18

https://kyso.io/vanaurum/algorithmic-portfolio-optimization

https://kyso.io/Don/a-journey-to-the-tip-of-neural-networks

https://kyso.io/waldohiding/twitter

Happy to hear any and all feedback!

/r/IPython
https://redd.it/8r2lsf

Python ANOVA using Statsmodels and Pandas
https://youtu.be/Lp2aV_4LF48

/r/pystats
https://redd.it/8r0evn

How to make a bouncing ball simulator in Python
https://boingboing.net/2018/06/13/how-to-make-a-bouncing-ball-si.html

/r/Python
https://redd.it/8r0mf3

Python Tutorial: 3D Wrecking Animation With 14 Lines of Code
https://www.youtube.com/watch?v=jXEW1MWkc18&feature=youtu.be

/r/Python
https://redd.it/8r0e5e

Recommend a scipy.stats.chisquare ad hoc test?

Hello again,
My scipy chi square test of independence returned a p-value of 0.000, I have age/gender groups 0-14 Male and 0-14 Female etc up to 95.

Should I apply an ad hoc test to the whole datatset, or should I break up the catergories and do the chi-square rest for each age/gender group? I.e. compare 0-14 M to 0-14F to test for independence?

Thanks

/r/pystats
https://redd.it/8r4rkr

Creating/Updating multiple object instances

I am trying to wrap my head around how to structure a form that will allow me to create/update a variable number object istances.

I currently have a view that dynamically generates budget line items based on a project number in the URL parameter. All budget items live in model (A).

In model (B), I need to allow the user to enter/update a number for each budget line item. All this needs to happen on the same page.

To visualize: https://i.imgur.com/PN58dRD.png

The *Fcst Hrs* column is where the user will enter/update items for model (B). Each line they're updating/creating will be a row in my DB. I just don't know how to structure this in django.

/r/django
https://redd.it/8r4ude

Decided to start learning Python last week, and using Python and Spotify's API, I created a playlist that's automatically updated from BestEverAlbums.com's Top tracks of all time list

Using a throwaway since I cannot change my Spotify username

The playlist is here: https://open.spotify.com/user/1285950058/playlist/0d8k1FfVCrCZUnSXTcAlUE

The playlist is generated from crawling the first 10 pages of the songs on this list: https://www.besteveralbums.com/topratedstats.php?o=track

The playlist is set to update every day, though I don't know how much this list changes over time.

There's some limitations, however:

* If the song isn't on Spotify, there's not much I can do about that.
* If the text of the song doesn't match the name of the song exactly on Spotify, the song won't get added to the playlist. (I'm trying to figure out a way to make the search results more accurate)
* I could not get the crawler to actually add the songs in order, so the order right now is random

Using this method, I was able to accurately add 84 out of the top 100 songs on this list.

I'd love to hear your feedback, and feel free to follow the playlist!

(Sorry if this breaks any rules on the subreddit)

/r/Python
https://redd.it/8r5dku

Kyso – Publish Your Jupyter Notebooks as Blog Posts

Hi Guys - the founders of kyso here.

Kyso (https://kyso.io) is a blogging platform for data science. We render your jupyter notebooks, including all code, interactive visualisations and rich markdown, as awesome data blogs.

Upload existing notebooks, import your repos from Github and start brand new Jupyterlab environments in the cloud, all for free.

And then publish and share your blogs with the community!

Have a look here for a few example posts on our platform:

https://kyso.io/jamesle/fifa18

https://kyso.io/vanaurum/algorithmic-portfolio-optimization

https://kyso.io/Don/a-journey-to-the-tip-of-neural-networks

https://kyso.io/waldohiding/twitter

Happy to hear any and all feedback!

/r/JupyterNotebooks
https://redd.it/8r2ilv