Python Daily

nbmerge: merge / concatenate Jupyter notebooks
https://github.com/jbn/nbmerge

/r/IPython
https://redd.it/6a5ele

GitHub - jbn/nbmerge: A tool to merge / concatenate Jupyter (IPython) notebooks

A tool to merge / concatenate Jupyter (IPython) notebooks - jbn/nbmerge

34 views17:30

4 Python libraries for building great command-line user interfaces
https://opensource.com/article/17/5/4-practical-python-libraries

/r/Python
https://redd.it/6a67l7

Opensource.com

4 Python libraries for building great command-line user interfaces

In the second installment of a two-part series on terminal applications with great command-line UIs, we explore Prompt Toolkit, Click, Pygments, and Fuzzy Finder.

42 views18:30

Python Daily

Hey Python/pandas users. Check this out. I've used this as part of my day-to-day workflow since it was released a couple months ago and I'm hooked. It's a better way to store, retrieve, and explore my data in a way that's seamlessly integrated with my preferred analysis environment (Python).

https://data.world/nrippner/explore-the-data-world-python-sdk/file/ddw_SDK.ipynb
or, to look at notebook without data.world account:
https://github.com/nrippner/misc/blob/master/ddw_SDK.ipynb

/r/pystats
https://redd.it/6a7rnd

data.world

41 views20:30

Python Daily

Requests per second with database query

I am trying to get a sense of performance based on my set up because I feel like I am way under performing.

I have a flask container on AWS ECS with 1024Cpu Units and 2GB of memory.

The containerized app uses uwsgi and nginx.

The query I am running is a sql-alchemy pagianted query with a page size of 20. Running this same query from SequelPro takes 109ms. This is against a remote Aurora RDS.

However when load testing this endpoint (with 2 containers under an ELB) the requests per second is hardly double digits. Is this just a symptom of having a remote db and small hardware profile on the container?

/r/flask
https://redd.it/6a79u3

Requests per second with database query • r/flask

I am trying to get a sense of performance based on my set up because I feel like I am way under performing. I have a flask container on AWS ECS...

32 views21:30

Python Daily

From the MachineLearning community on Reddit

Explore this post and more from the MachineLearning community

35 views22:30

Python Daily

Looking for resources for deserializing JSON into multiple models

I'm very new to both python and Django and am having difficulty finding the right documentation to work through my current problem.

I am trying to take JSON from a url and commit it to a database. For the example below I would like to commit "name", "type", and "level" to a model called Items and all of the details to a model called ItemDetails.

[
{
"name": "Abomination Hammer",
"type": "Weapon",
"level": 0,
"rarity": "Fine",
"vendor_value": 0,
"default_skin": 5014,
"game_types": [
"Activity",
"Wvw",
"Dungeon",
"Pve"
],
"flags": [
"NoSell",
"SoulbindOnAcquire",
"SoulBindOnUse"
],
"restrictions": [],
"id": 15,
"chat_link": "[&AgEPAAAA]",
"details": {
"type": "Hammer",
"damage_type": "Physical",
"min_power": 146,
"max_power": 165,
"defense": 0,
"infusion_slots": [],
"infix_upgrade": {
"id": 112,
"attributes": []
},
"secondary_suffix_item_id": ""
}
}
]

So far the only way I've found to do it is to load the JSON from the url

def jsonload(url):
response = urllib.request.urlopen(url).read()
jsonstring = str(response, 'utf-8')
json_load = json.loads(jsonstring)
return json_load

Then loop through each item (like the one shown above) and save the fields I want to use.

i = Items(id=data.get('id'),
name=data.get('name'),
type=data.get('type'),
level=data.get('level'),
)
i.save()

From what I've been able to gather it seems like this is better suited for a deserializer, but I can't find anything about deserializing JSON from a url into multiple model classes.

/r/django
https://redd.it/6a7mq5

Looking for resources for deserializing JSON into... • r/django

I'm very new to both python and Django and am having difficulty finding the right documentation to work through my current problem. I am trying...

34 views23:30

Python Daily

PyTree: Creating and drawing realistic trees!
https://github.com/PixelwarStudio/PyTree

/r/Python
https://redd.it/6a83dn

GitHub

GitHub - PixelwarStudio/PyTree: Package for generating and drawing (fractal)trees.

Package for generating and drawing (fractal)trees. - GitHub - PixelwarStudio/PyTree: Package for generating and drawing (fractal)trees.

37 views00:30

Python Daily

Pandas 0.20.1 Released
http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#v0-20-1-may-5-2017

/r/IPython
https://redd.it/6a70mn

Pandas 0.20.1 Released • r/IPython

8 points and 0 comments so far on reddit

42 views01:30

Python Daily

Using DjangoQL with default search bar

There's a neat library called [DjangoQL](https://github.com/ivelum/djangoql) which I want to incorporate into my project. It replaces the standard search_fields search bar in the Admin with a very powerful query system.

However, I want my user to be able to use the normal search_fields most of the time, and only use the DjangoQL bar on occasion when the query functionality is really needed.

Is there some way that I could have there be two search bars, one DjangoQL and one search_fields? Or even better, to have an "advanced search" button which toggles/pulls up/etc the DjangoQL search bar?

Thanks so much!

/r/django
https://redd.it/6a9w07

GitHub

GitHub - ivelum/djangoql: Advanced search language for Django

Advanced search language for Django. Contribute to ivelum/djangoql development by creating an account on GitHub.

45 views02:30

Python Daily

Populate form field from ModelChoiceField with previously selected value

I'm trying to let my users edit a form they have previously created. All my form fields are being populated with their existing values with one exception. I have a ModelChoiceField that I'm unable to populate, it always shows the none value ("Select Client"). So my question is, how to I get this field populated when editing a form?

https://pastebin.com/3ACrHYpW

/r/django
https://redd.it/6a9ggq

Pastebin

[Python] hellowebapptest - Pastebin.com

37 views03:30

Python Daily

An informal proof of Kleene's and Rice's theorems using Python
https://github.com/mapio/feeding-rice-to-the-snake

/r/JupyterNotebooks
https://redd.it/6a5ds4

GitHub

GitHub - mapio/feeding-rice-to-the-snake: An informal proof of Kleene's and Rice's theorems using Python

An informal proof of Kleene's and Rice's theorems using Python - GitHub - mapio/feeding-rice-to-the-snake: An informal proof of Kleene's and Rice's theorems using Python

39 views05:30

Python Daily

Deployment best practices for secret key, environment variables, etc

I just spent days - DAYS - on my first real AWS EC2 deployment, all because I was setting environment variables on the production server and apparently that doesn't work. I finally figured it out and was able to deploy with my key and passwords hard-coded into my settings.py on the production server which, I assume, is not something I should be doing? But then Django docs say to use env vars or to import from a file, but that sounds just as insecure as hard coding it into settings.py, so maybe I'm overthinking this.

What the standard practice in production? Where do you put the secret key and, say, the email server password? When I search for environment variables + AWS EC2 there's a whole bunch of results I don't understand involving startup scripts and whatnot, which I'm not using at this point.

/r/djangolearning
https://redd.it/6a7wfy

Deployment best practices for secret key,... • r/djangolearning

I just spent days - DAYS - on my first real AWS EC2 deployment, all because I was setting environment variables on the production server and...

39 views07:30

Python Daily

Speedml is a Python package to speed start machine learning projects.
https://github.com/Speedml/speedml

/r/Python
https://redd.it/6aa57b

GitHub

GitHub - Speedml/speedml: Speedml is a Python package to speed start machine learning projects.

Speedml is a Python package to speed start machine learning projects. - Speedml/speedml

44 views10:30

Python Daily

Looking for feedback on my first tutorial: How to deploy Django app on AWS (with Elastic Beanstalk)
https://codepal.herokuapp.com/deploying-django-app-on-aws-using-elastic-beanstalk/

/r/django
https://redd.it/6a6a1h

CodePal

How to deploy a Django app on AWS (with Elastic Beanstalk)

This tutorial is walk-through of how to deploy a Django application to AWS Elastic Beanstalk (with Postgres).

40 views11:30

Python Daily

Worth the switch to Pycharm?

I've been using Emacs for a while to do things like stats in R (ESS), organizing my life (org-mode), and writing papers (auctex). So I've got quite a bit of investment in Emacs. I'm not a super-user by any means, but I can use it pretty efficiently.

Now that I'm starting to do a lot more Python development, I've heard over and over again how amazing Pycharm is. I have access to the full edition, so I'll be able to use it to its full potential. I am wondering about a few things, though, as well as the general "is it worth it?" question.

* How good is IdeaVim? I use spacemacs in Emacs, so I'm used to bopping around inside and between files with vim-style shortcuts.
* Is there an easy way to sync settings between machines? I do development on 2-3 different computers (home, work, laptop), and I'd like to keep everything synced up between the three.
* What are the killer features I should switch for?

EDIT new question:

* Is it possible to have machine-specific settings in addition to syncing general settings? For example, my laptop is super small, so I usually use a slightly smaller font size there so I can see a bit more code.
* I also want to start using Python for more of my data analysis. Does Pycharm work well for that, or should I use a different package for the stats stuff?

Thanks y'all.

/r/Python
https://redd.it/6ac5pk

Worth the switch to Pycharm? • r/Python

I've been using Emacs for a while to do things like stats in R (ESS), organizing my life (org-mode), and writing papers (auctex). So I've got...

48 views13:30

Python Daily

I started building a Flask API service using Python 3.6 and I can't connect to the database. More info below

I have the following code for database connection:

from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()

In other part of the app I have some code for adding tables, etc.:

db.init_app(flask_app)
db.create_all()
db.session.commit()

This is the error I'm getting: `ModuleNotFoundError: No module named 'MySQLdb'

What's going on here? I'm positive I have `MySQL` on my Mac. Is this because I'm using Python 3?

Thanks in advance!

/r/flask
https://redd.it/6a35tn

I started building a Flask API service using Python 3.6... • r/flask

I have the following code for database connection: from flask_sqlalchemy import SQLAlchemy db = SQLAlchemy() In other part of the app I...

44 views15:30

Python Daily

MicroPython's WebREPL on the ESP8266
https://dev.to/kenwalger/micropythons-webrepl-on-the-esp8266

/r/Python
https://redd.it/6adm1j

DEV Community

MicroPython's WebREPL on the ESP8266

See how to implement a WebREPL interface on the NodeMCU ESP8266

38 views16:30

Python Daily

Project-based tables and Django

Hi everyone,
I am looking for some suggestion/input regarding how to structure the database for a web-based application I will need to develop in the following months.

Basically I will need to develop an application to organize technical information (cable connections, tags, I/O boards, etc). The main issue is that this documentation is project-based, and I'd like to avoid to have everything merged together (i.e. I don't want to have a single table with all I/O boards for all the projects, as I'll have to filter by project in any query I do, and the table size will quickly grow out of control).

My idea is to create a different DB everytime a new project is started, and to populate said DB with the required empty tables ("bootstrapping a project"). Then I'd like to access the various projects via URLs (like /myapp/project1/ will "open" project1).

Probably the easiest thing would be to just create a different instance of the web app for each project, but that would make more difficult the creation of a new project (instance a new container/VM/whatever for the app...), as well as "sharing data" functions (like "copy all this I/Os from project A to B"). So I'd like to avoid this path.

Is there a best practice for this, or do you have any suggestion from your experience? I know that this is a Django subreddit but... perhaps Django is not the best candidate for something like that, and is better to go with something more... barebone (e.g. flask)?

Thanks! :)

/r/django
https://redd.it/6adio8

Project-based tables and Django • r/django

Hi everyone, I am looking for some suggestion/input regarding how to structure the database for a web-based application I will need to develop in...

45 views18:30

Python Daily

Automate Everything with Python - Question

Hi Ya'll,

I'm learning Python from Automate the Boring Stuff. My first project is to download episodes of this podcast I like. Right now when I run the program is downloads all the HTML from the 'url.' Where am I going wrong?

Next question. What it is supposed to do next is Click on all the links that say "Download this Episode." I have that bit of code included in the below but it doesn't seem to be working yet. Also, any tips for someone learning how to work with python would be appreciated!

Below is the python script:

https://drive.google.com/open?id=0B_w5K7I4N0d3a0JReW1QRUNPUXM

/r/Python
https://redd.it/6ad823

36 views19:30

Python Daily

Using PreFetch with DRF ViewSets to include a @property on a model that relies on reverse foreign keys.

It took me entirely too long to figure this out, so thought I would share in case others are running into the same thing. Note that I attempted to use a OneToOne, but because not every Model B can relate to a Model A, and it throws duplicate errors on Nulls... I couldn't.

Model B has a FK to Model A.
Model A has a @property to obtain a field from Model B from a single model instance. In this scenario, I know there will only be one, so I go get the first value only.

@property
get_model_B_field(self):
b = self.modelB_set.values('field').all()
if b:
return b[0].field

Of course, including this in the serializer quickly creates the N+1 problem, as Django will re-query the database for each Model A in the queryset. As I have several properties like this across a variety of models, the problem grew exponentially until I had thousands of queries running and 12+ seconds to load the api page. P.S. I highly recommend using [Django Debug Toolbar](https://django-debug-toolbar.readthedocs.io/en/stable/).

First attempt was to use prefetch_related as such in the ViewSet which calls the serializer:

queryset=ModelA.objects.prefetch_related('modelB_set').all()

Still N+1

I went back to the property and attempted to remove the "values" aspect, and also make use of the first() function.

@property
get_model_B_field(self):
b = self.modelB_set.first()
if b:
return b.field

Still N+1

Then I tried using the new Prefetch object on my view's queryset, as such:

queryset=ModelA.objects.prefetch_related(Prefetch('modelB_set',queryset=ModelB.objects.all()))

Still N+1

Finally, I read the bug report from another user indicating first() wasn't respecting the prefetch, to realize that in order for that to happen, you must explicitly set an order on the queryset in the PreFetch()

queryset=ModelA.objects.prefetch_related(Prefetch('modelB_set',queryset=ModelB.objects.order_by('field').all()))

Finally, down to 7-8 (from 5000+) queries for a rather large API returning data from across multiple models, many of which are reverse FK lookups in about 1 second. If others have a more elegant solution, I'd love to hear it. My only other option was going to be attempt to use Nested Serializers, but I didn't want to nest all of ModelB inside ModelA, and I didn't like the idea of creating multiple serializers against ModelB to just handle different field combinations. Now, I can have all my properties in one place (the model), and include the fields individually on the API as needed.

/r/django
https://redd.it/6aexjt

42 views20:30

About

Blog

Apps

Platform