Content scraping with Selenium

Hi

I am building a site that allows users to enter a URL with the aim of it grabbing the images from that site and allowing the user to pick one to use on their page. Standard concept I imagine. I have it all working, along with some code that checks the file size to make sure it's above a certain size. However it's slow, for instance running this on my local machine for a given site may take 12seconds, doing the same on a site with similar functionality takes 3 seconds. Even if I disable the portion of my code that determines the file size it's still as slow.

Is there a way to optimise this? I notice the site that I compared mine with only grabbed 5 images whereas mine took about 20+.

Can I tell selenium to only run for maybe 3 seconds and then use whatever it grabs within that time frame? Or anything else that would help speed this up in any way. I would be happy to only expend as much energy as is required to take the first 5 images from the page - some of the pages that will be scraped by users will be really large I imagine.

Thanks in advance for any assistance / pointers.

/r/django
https://redd.it/8hz6p6

[AF]Flask + Multicore: will flask use all cores automatically?

I have very little knowledge in this area but I'm curious. If I write a Flask application and deploy it onto a virtual machine with multiple cores, then will my Flask application be able to utilize all cores out of the box? If not, then can it be configured to (conceptually)?


/r/flask
https://redd.it/8gafpl

Anyone have a spare ticket they could sell for euroDjangoCon this month? I'm keen to go, but just found out about it....



/r/django
https://redd.it/8i46xe

Thumbnail for file fields

Hey guys,

Has anyone come up with an effective way to display image thumbnails within an ImageField (or VersatileImage) field in Django after the user uploads the image? I see that there are a few code snippets out that that handles thumbnail images for the admin area, but I'm interested in displaying the thumbnail of a user uploaded image on my web page. I'd imagine that I'd need a custom widget, but I don't know much about widgets or how to implement them. Any thoughts would be greatly appreciated.

Thanks!

/r/django
https://redd.it/8i4agq

Advice on database topology? visualizer

Hi all, I'm looking for advice as to the approach I should take.

I have a database something like this;

Animal
Mammal Fish
Cat Dog Cod Plaice

And I'm just wondering of a *sexy* way to visualize the data using a django approach where you can explore along the options tree branches and see the Cat at the end of the tree. I've used django a fair bit before.

Wonder if anyone has any advice or can point me in any direction of good platforms/github repos I could copy and paste my database into and tweak? Something ideally relatively secure such that I could put onto the web at the end.

/r/django
https://redd.it/8hxkfy

How to write a command-line application using Python and Click
http://progogo.work/how-to-write-a-command-line-application-using-python-and-click

/r/Python
https://redd.it/8i48un

Django Vs Rails

hi

i am interested in learning a framework for web. i have read that Django is more modern but rails is fast. it is not clear which one i should learn. rails has been around for a while now and people are saying it is dying. is this true?

i would love to know your opinions

/r/django
https://redd.it/8i5b0y

Django Tutuorial 03: More Views, and CSS
https://youtu.be/MSYcVKUy_eU

/r/djangolearning
https://redd.it/8i2kg1

Help with wtforms and socketio/session data

Hi, I’m new to flask and web development. I’m building my first web app with flask and loving it, learning a lot. My web app allows people to join groups and chat with each other on the specific group page. Wondering what is a good way to pass the username back to client with the message? Or should I put the username in the html (I have it as a variable in the view function), and then try to access it in my jquery script?

Also, I have a basic question about wtforms classes. I am currently using the Form class but am interested in the FlaskForm class. It seems from what I’ve read that FlaskForm offers more. Just interested in the difference/benefits of either.

/r/flask
https://redd.it/8gacsp

Post your Flask questions : Miguel Grinberg's Flask Webcast is LIVE
https://www.youtube.com/watch?v=Z4X5Oddhcc8

/r/flask
https://redd.it/8dwyio

Building a notification system part 3: Dropbox
https://smetj.net/building_a_notification_system_part_3_dropbox.html

/r/Python
https://redd.it/8i85jj

Air pollution analysis with pandas

Hi,
I have started a project where I try to analyse air pollution data from monitoring sites in Munich, Germany. Anybody knows of publicly accessible air quality analysis using python/pandas? My initial scripts:

https://github.com/jsln/aq-sensor-data-analysis

I want to extract as much information as I can from the samples before attempting to use scikit-learn to:

- Compare the performance of different forecasting models.
- Study correlations between samples at different monitoring stations.

Any work in this area you can point me at, I would appreciate it.

Thanks,
Juan.

/r/pystats
https://redd.it/8i8fhe

[Code Review] I'm working on a vinyl trading API and it's currently WIP but I'd like some feedback.

Link: https://github.com/mabbie/vinylhandelsclub-api

I still have ways to go before the project is complete, mainly security and handling logging in and out users. But so far, how am I doing? I used Flask-RESTful to build the API, and Flask-JWT-Extended for JWT authentication.

/r/flask
https://redd.it/8dzhyb

Humble Bundle Python dev kit - charity
https://www.humblebundle.com/software/python-dev-kit-bundle?hmb_source=humble_home&hmb_medium=product_tile&hmb_campaign=mosaic_section_1_layout_index_2_layout_type_twos_tile_index_2

/r/django
https://redd.it/8i5po0

Is anybody doing qualitative data analysis in Jupyter / IPython?

For R there is the RQDA package and a number of other tools related to qualitative data analysis. There are very few IPython notebooks on the web that deal with qualitative sociological approaches, but most of them are undocumented.

Therefore my question: is anybody doing a qualitative project in a Jupyter notebook? Would you like to present it here and post a short description of what it is about?

/r/IPython
https://redd.it/8i3r72

[R] Holy shit you guys, the new google assistant is incredible.
https://youtu.be/pKVppdt_-B4

/r/MachineLearning
https://redd.it/8i3zll

Announcing glom: Restructured Data for Python
https://sedimental.org/glom_restructured_data.html

/r/Python
https://redd.it/8i97g5

How to "visit" a page without "visiting" it in Flask?

Okay, so I might just be confused as fuck or something.

When I 'visit' a page that triggers a route on my app, say like '/install', that route executes and redirects to another route, eventually generating an auth code.

Now, I want to run this route, without having to visit the page - so basically, I am trying to run this route so that I can get the auth code in the end.

Now I am calling this route from an external website. I am simply sending a GET request to a function 'load' and in that function there is another function called: get_install(), which contains the below code.

call = requests.get(url_for('install', _external = True, _scheme = "https"))

Now this code works but when I type print (request.args) nothing comes up.

So I am wondering, how do I execute the install route without visiting the {my_app_url}/install route?


/r/flask
https://redd.it/8dyr7z

Creating Dynamic Forms with Django
https://www.caktusgroup.com/blog/2018/05/07/creating-dynamic-forms-django/

/r/django
https://redd.it/8ic8u3

Custom module not running in Notebook

Hi all,

Say I have the following in my notebook:

// some code
// bunch of other code

It runs perfectly, but if I try to create a bunchofothercode.py file and put // bunch of other code in it, and then:

// some code
import bunchofothercode

This doesn't give any errors, but it doesn't run the code.

I'm sure I am missing something, but I don't know what :)

/r/IPython
https://redd.it/8id40o