[AF] advice about streamlining long running processes

Hi,

I made a Python program that processes large data files. I was asked to make the program accessible through a web interface. I would like to use Flask for this, but i have no experience with Flask yet.

The program flow would be that my colleagues can upload a file and set parameters for the processing > add the job to a queue > process the job > send an email when the job is done > they can download the processed file.

Processing the job is usually done between 5 -15 minutes. What is the preferred way to set this up? I'm worried that because of to the processing time, the website would freeze up and becomes unresponsive.
Is it better to handle the processing separate from my Flask application?

Any advice is greatly appreciated.

/r/flask
https://redd.it/7pn0rd

How to pass an already defined function to "@app.rotue("path")"?

I'm not super good with decorators but I kinda know how they work, but can't get it work.

so I don't want to define my functions and routes in the same file.


@app.route("/")
def home():
return render_template("home.html")

I wanna seperate those in different files.
I thought I can do


def home():
return render_template("home.html")

then

from .controller import home
app.route("/", home)

but it does not work

I hope my question make sense.


/r/flask
https://redd.it/7oyj7t

Retrieving lat/long coordinates from the client without requiring a form POST

Is it possible to retrieve the lat/long coordinates immediately after the client "allows" access to their location?

Currently upon agreement to share their location, we'll have access to their coordinates, but only in the browser. How would I send this information server-side ASAP in order to do some processing?

Essentially, how do I skip the POST step after the user agrees to share their coordinates?



/r/django
https://redd.it/7pli92

Co-Worker & Philosophical Differences

I am curious how others have dealt with co-workers who may have had fundamental difference in best practices with co-workers in the past.

Some context:

I have a co-worker (with several years of seniority at this job) who looks at things quite differently than I. While we both agree on KISS, our perspective on what is simple is completely different. We work on primarily "web-applications" but tend to integrate with a lot of external systems.

Some examples:

- The co-worker wants to keep the dependencies as minimal as possible, almost without exception. I agree with not adding complexity needlessly, but am also opposed to reinventing the wheel on the job. Example: limiting ourselves to a document store when the data is very relational in nature. On my end, this adds significant code complexity to perform joins, Cascade deletes, verify relationships are still valid, etc... While data can be structured in a way that makes sense here in some cases, the nesting became too deep for Mongo.

- When dealing with code that would run slower than is acceptable in a single request/response cycle, I would tend to push the logic to Celery, use caching and/or other techniques along those lines. My co-worker does things like spin off a thread from the gunicorn worker or replicates an entire large, external database.

- The co-worker isn't remotely open to the idea of using ORMs on principle. The claim is that you can't optimize if needed (the truth to that statement can be debated) and it adds additional complexity co-worker has to wrap their brain around. However, our applications tend to be small, our deadlines short, and maintainability has been our largest issue historically. I think introducing an ORM (e.g. sqlalchemy to avoid including Django and stay somewhat slim) would totally work for us, significantly decrease the time to build out MVPs, make our code easier to reason about, and consequently make it easier to maintain our applications in the long-run.

- There are times we needed a cache. The co-worker absolutely refuses to inclide Redis (as a counter-point they have no problem with using elastic search for searching/indexing text) on principle of staying slim. We have historically used Mongo as our primary DB so I wrote a cache decorator to compare Mongo vs. Redis. Mongo is not accurate with TTLs, but more importantly Redis was returning results in a 10th of the time. This, to me, is a case of using the right tool for the job.

- There is background processing we do in several applications. A previous co-worker wrote a distributed cron-runner (avoids running the same job on multiple servers simultaneously) which is not a bad solution. However, these tasks sometimes need to be kicked off when a request is made. I think Celery and Celerybeat is a great tool to fit this job; again, that idea isn't well received.

There are other examples of being given unsolicited feedback on even the most minor design decisions. I can't tell if the co-worker absolutely thinks I am incompetent, if they think on a different wavelength, or if I am missing something important. I've been at this for around 11 years but only 1 with the current company. I am not coming from a perspective of trends or laziness but adapting my previous experience on many, many projects with the needs of my current team. I am open to the idea that co-worker knows many things I don't, but the constant resistance to "everything" is making it very difficult to work with them and run my own projects successfully. I do wish I could understand co-worker's perspective better but I am at a loss.

Anyways, I just wanted to see if anyone else has ran into times where they had to work with someone who was very opinionated, it felt like they were not empowered, and how they handled it. I am asking here, because in the end 90% of our work is writing Python applications.

/r/Python
https://redd.it/7pngyz

Tired of logging to a terminal? I made a GUI for you!

I just released [cutelog](https://github.com/busimus/cutelog), a GUI for Python's standard logging module. It looks [like this](https://i.imgur.com/LQi8ccg.png).
It's cross platform and requires only minimal changes to your code (if you're already using the logging module).

It can do most things you would expect: searching, filtering, multiple connections, customization of the output table, limiting number of records per connection, merging records of several tabs. There's also a [dark theme](https://i.imgur.com/BGUqHM2.png)!

It can also do things you would _not_ expect, because it's in beta and may have bugs. Submit an issue on GitHub if you find one of those!

This is my first released project of any kind, so any feedback is welcome.

/r/Python
https://redd.it/7ppfph

Data in object is updated and saved in AJAX, but have to refresh view to see new data.

So, I have an AJAX call on `submit` which takes some data and updates an object and I save that object, where then it will automatically do the POST section in the same view which saves everything and redirects me to a `detail` view which shows me everything pertaining to the whole object. The data I updated is not changed until I refresh the page...but this only happens 50% of the time. It is almost like the `detail` view is occurring before the object saves which doesn't make sense because the redirection is the last line of code before switching to that view.

/r/djangolearning
https://redd.it/7pqt5k

A 100% python tutorial on working with smart contracts and decentralized applications

A couple weeks ago I posted https://www.reddit.com/r/Python/comments/7lr48g/a_minitutorial_on_deploying_smart_contracts_with/

At the time I posted the tutorial it was about 50/50 javascript and python. I've updated it to be 100% python.

I tried to make it as minimal as possible so it didn't distract from the lesson on decentralized applications. More importantly I try to keep this repo updated to the most current versions of web3py (python module used to interact with smart contracts) and solidity (the language smart contracts are commonly written in). The smart contract world changes super fast and I'll do my best to keep this current!

Thank you!

/r/Python
https://redd.it/7ppv29

My first Twitter bot just made its debut post! I've made a few bots, but this is the one I'm most proud of.

Finished product first:

https://twitter.com/DSCOVRDaily/status/951369376451592192

DSCOVRDaily is a pretty simple bot that takes the images of the earth from [here](https://epic.gsfc.nasa.gov/) and makes a daily GIF. It's been posting for a while to [Imgur](https://imgur.com/user/DSCOVRImagesDaily/submitted), but I thought I'd try playing with Twitter for the first time.


/r/Python
https://redd.it/7pmvku

The Square-Sum Problem - Numberphile

[The Square-Sum Problem - Numberphile](https://www.youtube.com/watch?v=G1m7goLCJDY)

*"Given the digits 1..15 arrange them so that all adjacent numbers add to a square number"*





/r/Python
https://redd.it/7prgyg

Home Advantage in Football Leagues Around the World
https://dashee87.github.io/data%20science/python/home-advantage-in-football-leagues-around-the-world/

/r/pystats
https://redd.it/7psjyi

How to get friendly Validation Errors

I have the following model and have a nice, friendly validation error message in a custom clean method that works great, it provide a message and a link to follow. However, I also need a custom save method which then causes the validation error to show up as a debug error when I call clean. How would I achieve a friendly validation error here?

FYI: this is an old, old legacy server running django 1.3, python 2.7

class Event(models.Model):
title = models.CharField(max_length=100)
slug = models.CharField(max_length=50, help_text='Automatically built from the title, location and start time.', default="DO NOT EDIT")
location = models.CharField(max_length=50)
event_start_time = models.TimeField()
...
# clean method that will return a friendly error if the slug is not unique and provide a link to list other records with the same slug:
def clean(self, *args, **kwargs):
from django import forms
from django.utils.safestring import mark_safe
while Event.objects.filter(slug=self.slug).exclude(pk=self.pk):
error_message = 'Please double check that your event has not already been entered into the calendar. An event with the exact or very similar title, location and start time already exists.<br /><br /><a href="/admin/events/event/?q=%s">Click here to check existing matching records</a>.' % self.slug
raise forms.ValidationError(mark_safe(error_message))

def save(self):
from django.template.defaultfilters import slugify
self.slug = "%s-%s-%s" % (slugify(self.title)[:39], slugify(self.location), slugify(self.event_start_time)[:4])
self.clean()
return super(Event, self).save()


/r/django
https://redd.it/7psad5

gifmaze: make GIF animations of various maze generation and maze solving algorithms
https://github.com/neozhaoliang/gifmaze

/r/Python
https://redd.it/7puvuh

Created a bot that combines the most popular news images

/r/Python
https://redd.it/7prz39

[D] Introduction to Various Reinforcement Learning Algorithms. Part I (Q-Learning, SARSA, DQN, DDPG)
https://medium.com/@huangkh19951228/introduction-to-various-reinforcement-learning-algorithms-i-q-learning-sarsa-dqn-ddpg-72a5e0cb6287

/r/MachineLearning
https://redd.it/7pwinl

[AF] Newb question: Can't get Flask-SocketIO and Flask-SSLify to work?

I generated a self-signed certificate using OpenSSL following the procedure found online, saving the certificate and key as "cert.pem" and "key.pem". This is my minimal test case:

from flask import Flask
from flask_socketio import SocketIO
from flask_sslify import SSLify

app = Flask(__name__)
app.secret_key = "secret!"
socketio = SocketIO(app)
SSLify(app)

@app.route("/")
def home():
return "Hello world!"

socketio.run(app, host='0.0.0.0', port=80, certfile='cert.pem', keyfile='key.pem')

Then when I try to visit my site, I get this error:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/eventlet/wsgi.py", line 750, in process_request
proto.__init__(sock, address, self)
File "/usr/lib/python3.5/socketserver.py", line 681, in __init__
self.handle()
File "/usr/lib/python3.5/http/server.py", line 422, in handle
self.handle_one_request()
File "/usr/local/lib/python3.5/dist-packages/eventlet/wsgi.py", line 356, in handle_one_request
self.raw_requestline = self.rfile.readline(self.server.url_length_limit)
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.5/dist-packages/eventlet/green/ssl.py", line 204, in recv_into
return self._base_recv(nbytes, flags, into=True, buffer_=buffer)
File "/usr/local/lib/python3.5/dist-packages/eventlet/green/ssl.py", line 225, in _base_recv
read = self.read(nbytes, buffer_)
File "/usr/local/lib/python3.5/dist-packages/eventlet/green/ssl.py", line 139, in read
super(GreenSSLSocket, self).read, *args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/eventlet/green/ssl.py", line 113, in _call_trampolining
return func(*a, **kw)
File "/usr/lib/python3.5/ssl.py", line 791, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.5/ssl.py", line 575, in read
v = self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:1977)

Am I missing something basic here? A configuration setting? Or am I using the extensions wrong? SSLify is supposed to automatically redirect HTTP requests to HTTPS, right? I tried googling a bit, but it looked like I was doing everything according to the instructions...

(If it matters, Flask-SocketIO is running on eventlet, which worked fine before adding SSL)

/r/flask
https://redd.it/7pue5g

How To Create A Twitter Bot With Python
https://youtu.be/MN_1wOxIfRU

/r/Python
https://redd.it/7pwe38

[AF] Odd Syntax Error when trying to deploy app with apache and mod_wsgi

I’m learning about deploying flask apps using apache2.4 and mod_wsgi on an Unbuntu 16.04 server by building the simplest of ‘Hello, Word!’ apps, but I encountered a weird syntax error related to my wsgi file and I’m stumped.

Apache seems to be running fine but when I hit the app’s domain, I get apache’s standard 500 error and the logs list the following:

[wsgi:error] [pid 28427:tid 140540431517440] mod_wsgi (pid=28427): Target WSGI script '/var/www/myapp.domain.com/myapp.wsgi' cannot be loaded as Python module.
[wsgi:error] [pid 28427:tid 140540431517440] mod_wsgi (pid=28427): Exception occurred processing WSGI script '/var/www/myapp.domain.com/myapp.wsgi'.
[wsgi:error] [pid 28427:tid 140540431517440] Traceback (most recent call last):
[wsgi:error] [pid 28427:tid 140540431517440] File "/var/www/myapp.domain.com/myapp.wsgi", line 5, in <module>
[wsgi:error] [pid 28427:tid 140540431517440] exec(file_.read(), dict(__file__=activate_this))
[wsgi:error] [pid 28427:tid 140540431517440] File "<string>", line 4
[wsgi:error] [pid 28427:tid 140540431517440] deactivate () {
[wsgi:error] [pid 28427:tid 140540431517440] ^
[wsgi:error] [pid 28427:tid 140540431517440] SyntaxError: invalid syntax

The syntax error referenced in the logs are pointing to code in my virtual environment’s activation script located at ./venv/bin/activate

I’m wondering if my use the venv module vs. virtualenv is causing this issue

Here is the content of my app’s wsgi file (test is the name of the module were flask is instantiated):

#!/usr/bin/python3
activate_this = '/var/www/myapp.domain.com/venv/bin/activate'
with open(activate_this) as file_:
exec(file_.read(), dict(__file__=activate_this))

import sys
sys.path.insert(0, '/var/www/myapp.domain.com')

from test import app as application

Any guidance would be most appreciated.



/r/flask
https://redd.it/7pstd5

this buggy Python3 floor division

/r/Python
https://redd.it/7py9va

Python in 3D game development?

I am learning python on codecademy (Stop laughing for a second, I only knew some rookie level of HTML+CSS, I am honestly really impressed after spent a night on codecademy. Now I am able to read some stuff on github that was total alien to me.)

Now after some researches, most evidences convinced me that python is ..the least ideal language in game making sector for various reasons. But anyway it is *almost* non-existent in the industry, yeah I am aware that eve and some other was written in python. However, I also knew that there some outdated plugins and Panda3D could be used for games and also unreal engine community seem to have a ongoing plugin that allow user to give command in python.

My end goal of learning python is game prototyping. I wanted to be a game designer and came from media-art background doesn't seem to help much in job seeking. So I decided to learn python so I could prototype some stuff for portfolio purpose. I stalked on panda3D forum for a while, definitely saw some impressive things. On the other hand, the community seems only at hobbyist level which have me worried. If I go on apply for jobs, will python-based portfolio get rejected given that I am not willing to learn c++? (who has lifespan for that???)

/r/Python
https://redd.it/7pwniu

Humble Book Bundle: Python by Packt ends in 3 days
https://www.humblebundle.com/books/python-by-packt-book-bundle?partner=indiekings

/r/flask
https://redd.it/7q1pmp

[AF] How should two Flask websites run together on an Ubuntu server?

I have two Flask websites. One of them is a link-shortener and the other is a simple web file browser. I want them both to run on the same Ubuntu server. What would be a good, easy way to do this?

Specifically, the link-shortener ([ovipositor](https://github.com/wdbm/ovipositor)) saves to a database file and the web file browser ([a fork of browsepy](https://github.com/wdbm/browsepy)) can display file listings at some shared directory. They are for production (so should be someway secure) but I'm not worried about performance.

/r/flask
https://redd.it/7ps58l