plotting multiple sampled data points from a log

I have a log file which I need to plot in python with different data points as a multi line plot with a line for each unique point , the problem is that in some samples some points would be missing and new points would be added in another, as shown is an example with each line denoting a sample of n points where n is variable:

2015-06-20 16:42:48,135 current stats=[ ('keypassed', 13), ('toy', 2), ('ball', 2),('mouse', 1) ...] 2015-06-21 16:42:48,135 current stats=[ ('keypassed', 20, ('toy', 5), ('ball', 7), ('cod', 1), ('fish', 1) ... ]

in the above 1 st sample 'mouse ' is present but absent in the second line with new data points in each sample added like 'cod','fish'

so how can this be done in python in the quickest and cleanest way? are there any existing python utilities which can help to plot this timed log file? Also being a log file the samples are thousands in numbers so the visualization should be able to properly display it.

Interested to apply multivariate hexagonal binning to this and different color hexagoan for each unique column "ball,mouse ... etc". scikit offers hexagoanal binning but cant figure out how to render different colors for each hexagon based on the unique data point

extract comments and hyper links from cells of *.xls file using xlrd library in python

I want to extract comments and hyperlinks from cells of *.xls file using xlrd library in python in windows. Please let me know the functions and sample code.

Thanks in advance.

call python from matlab2015a

As matlab2015a has been able to support python. But it seems it can only call standard library in matlab. If I want to import other library such as numpy, scipy or sklearn, what should I do? And can I execute the python script directly. Unfortunately, the offical document of matlab has not given enough explanations. If anyone can explain, I will be really appreciated!

Python programming for card game War

This program is supposed to play the card game War. Okay so I need to be able to prompt the user to keep playing and have them respond by hitting the enter key. I'm not sure how to make this happen. Help?

import cards

# Create a deck of cards
the_deck = cards.Deck()

# Shuffle the deck, then display it in 13 columns
the_deck.shuffle()
print( "===== shuffled deck =====" )
the_deck.display()


def main():
    '''This function deals out half the deck to each
    player. It sorts the rank so as to solve the problem
    with Aces causing an infinite game'''
    player1_list=[]
    player2_list=[]
    for i in range( 26 ):
        p1_temp= the_deck.deal()
        player1_list.append( p1_temp )
        p2_temp= the_deck.deal()
        if (p2_temp.rank()==1):
            player1_list.append(p2_temp)
            player2_list.append(player1_list.pop(0))
        else:
            player2_list.append(p2_temp)


    print()
    # Card dealt to Player #1
    player1_card = player1_list.pop( 0 )
    print( "===== player #1 =====" )
    print( "Card dealt to player #1:", player1_card )
    print( player1_list )
    print()
    #Card dealt to Player #2
    player2_card=player2_list.pop(0)
    print( "===== player #2 =====" )
    print("Card dealt to player #2:",player2_card)
    print( player2_list )

    # Compare the two cards using overloaded operators
    print()
    if player1_card == player2_card:
        print( "Tie:", player1_card, "and", player2_card, "of equal rank" )
    elif player1_card > player2_card:
        print("Player #1 wins:",player1_card,"of higher rank than",player2_card)
    else:
        print("Player #2 wins:",player2_card,"of higher rank than",player1_card)
        print()
main()
def keep_playing():
    '''Determines whether the player wants to continue. If so they press the
    Enter key and the function calls main to start all over.'''
    still_playing=input('Press "Enter" to continue playing')
    Enter=1
    while still_playing==Enter:
        main()
keep_playing()

Error import spatial data in GeoDjango - KeyError for mpoly field

I was following the tutorial on http://ift.tt/1NnlWfX for setting up GeoDjango on my machine. But it seems like there is some issue there. While importing data using LayerMapping using load.run() from python shell, I get the following error:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/ubuntu/src/django/world/load.py", line 23, in run
    lm = LayerMapping(WorldBorder, world_shp, world_mapping, transform=False, encoding='iso-8859-1')
  File "/home/ubuntu/Envs/vir-env/local/lib/python2.7/site-packages/django/contrib/gis/utils/layermapping.py", line 105, in __init__
    self.check_layer()
  File "/home/ubuntu/Envs/vir-env/local/lib/python2.7/site-packages/django/contrib/gis/utils/layermapping.py", line 178, in check_layer
    ogr_field_types = self.layer.field_types
  File "/home/ubuntu/Envs/vir-env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/layer.py", line 153, in field_types
    for i in range(self.num_fields)]
KeyError: 12

Then I found out that, there is no 'MULTIPOLYGON' field in the .shp file:

>>> from django.contrib.gis.gdal import DataSource
>>> ds = DataSource('world/data/TM_WORLD_BORDERS-0.3.shp')
>>> layer = ds[0]
>>> layer.fields
[u'FIPS', u'ISO2', u'ISO3', u'UN', u'NAME', u'AREA', u'POP2005', u'REGION', u'SUBREGION', u'LON', u'LAT']

So, definitely in the world_mapping file, importing will fail for the 'mpoly': 'MULTIPOLYGON' mapping. Has anyone else faced this issue? I hope so, as I've followed the tutorial step-by-step. But it doesn't say anything about such issue.

Here's my load.py file:

  1 import os
  2 from django.contrib.gis.utils import LayerMapping
  3 from models import WorldBorder
  4
  5 world_mapping = {
  6     'fips' : 'FIPS',
  7     'iso2' : 'ISO2',
  8     'iso3' : 'ISO3',
  9     'un' : 'UN',
 10     'name' : 'NAME',
 11     'area' : 'AREA',
 12     'pop2005' : 'POP2005',
 13     'region' : 'REGION',
 14     'subregion' : 'SUBREGION',
 15     'lon' : 'LON',
 16     'lat' : 'LAT',
 17     'mpoly' : 'MULTIPOLYGON',
 18 }
 19
 20 world_shp = os.path.abspath(os.path.join(os.path.dirname(__file__), 'data/TM_WORLD_BORDERS-0.3.shp'))
 21
 22 def run(verbose=True):
 23     lm = LayerMapping(WorldBorder, world_shp, world_mapping, transform=False, encoding='iso-8859-1')
 24
 25     lm.save(strict=True, verbose=verbose)

Xpath extract dates between certain characters AND use as dates

UPDATE: Regarding my 2nd question (how to convert string to date format in MySQL), I found a way and want to share it:

1) Save the "string date" data as VARCHAR (Don't use TEXT)

2) When showing MySQL data in PHP or other ways, use the function of str_to_date(string-date-column, date-format), such as the following example:

$sql = "SELECT * FROM yourtablename ORDER BY str_to_date(string-date-column, '%d %M %Y')";

---

I am using scrapy to collect data, write to database. From a website, the post date of each item is listed as following:

<p>   #This is the last <p> within each <div>
<br>
[15 May 2015, #9789]
<br>
</p>

So the date is always behind a "[" and before a ",". I used the following xpath code to extract:

sel.xpath("p[last()]/text()[contains(., '[')]").extract()

But I will get the whole line:

[15 May 2015, #9789]

So, how to get only the part of "15 May 2015"? If this can be done, how to convert the scraped string (15 May 2015) as real DATE data, so it can be used for sorting? Thanks a bunch!

How to save a dataframe as a csv file with '/' in the file name

I want to save a dataframe to a .csv file with the name '123/123', but it will split it in to two strings if I just type like df.to_csv('123/123.csv').

Anyone knows how to keep the slash in the name of the file?

python 3.4.3 help: programming calculator with no buttons (entering entire expression)

I am creating a python program and need some help. Essentially I am making a calculator that solves the entire expression (ex. user types in 2 + 2). The program will read the equation the user entered as a string and manipulate the string in order to figure out the operands and the operator of the equation. I cannot use eval() in this script. So here is what I have written so far:

import operator

operList = ["+", "/", "%", "//", "**", "-",""," "]
Error1 = "Something is wrong with your equation: there is nothing in it."
Error2 = "Something is wrong with your equation: it does not feel complete."


def printGreeting():
    print("Hello, welcome to Equation Calculator")
    print("Enter in the expression that you want evaluated and the program will do the rest.")

def getExpression():
    userExp = input("Enter mathematical expression here: ")
    if not userExp.isdigit() or userExp.isalpha():
        print(Error2)
    elif operList[6] or operList[7] in userExp:
        print(Error1)
    else:
        return userExp

def add():
    if operList[0] in userExp:
        userExp.add()




printGreeting()
userExp = getExpression()

As you can see I am stuck on the function call def add(). My question is: Lets say the user entered 2 + 2, how would I go about evaluating the expression? I figured out how the function can recognize the operator + in the userExp but what next?

Note: I am very inexperienced with Python (just started a few weeks ago) and I really want to understand how to do this and any help is appreciated

Jython Type Error - Maximo automation script

I'm using the below script to see if 'ttype' is changed so I can reset the value of 'tclass'

if ttype != None & ttype != '':
  tclass=None

However I'm receiving the following error when triggering the script in the application:

TypeError: unsupported operand type(s) for &: 'NoneType' and 'unicode' in <script> at line number 1

I'm new to python/jython so any input would be helpful

How to make loop in python?

(1) Basically what I want to do is I want to extract the the links that contain this ("h3", "post-title entry-title") attributes for few pages in one website.

import urllib2
from bs4 import BeautifulSoup

# openlinkfile from texfile
links_file = open("link.txt")
links = links_file.readlines()          

for url in links:

# main page (read link from links)
htmltext = urllib2.urlopen(url).read()
soup = BeautifulSoup(htmltext)

# extract link in Links
relative_tags_to_desired_links = soup.find_all("h3", "post-title entry-title") 

for tag in relative_tags_to_desired_links:
    desired_element = tag.next_element.next_element
    print desired_element.get("href")

(2) To move to the next page, I will ask my code to read the the html codes that contain ("a","blog-pager-older-link") attributes and then will repeat the same thing as in (1)

  # page 2 (read link from links)
relative_tags_to_desired_linksp2= soup.find_all("a","blog-pager-older-link") 
for tagp2 in relative_tags_to_desired_linksp2:
    desired_elementp2 = tagp2.get("href")

    # read link from elementp2
    htmltextp2 = urllib2.urlopen(desired_elementp2).read()
    soupp2 = BeautifulSoup(htmltextp2)

    # extract link in elementp2
    relative_tags_to_desired_linksp2 = soupp2.find_all("h3", "post-title entry-title") 
    for taggingp2 in relative_tags_to_desired_linksp2:
        desired_elementp22 = taggingp2.next_element.next_element
        print desired_elementp22.get("href")

Let say I want to read 3 pages from one particular website, do I need to repeat the same codes until 3 times? How to simplify this code? I have tried few solution but still couldn't find any solution yet.

This is my code:

import urllib2
from bs4 import BeautifulSoup

# openlinkfile from texfile
links_file = open("link.txt")
links = links_file.readlines()          

for url in links:

# main page (read link from links)
htmltext = urllib2.urlopen(url).read()
soup = BeautifulSoup(htmltext)

# extract link in Links
relative_tags_to_desired_links = soup.find_all("h3", "post-title entry-title") 

for tag in relative_tags_to_desired_links:
    desired_element = tag.next_element.next_element
    print desired_element.get("href")


# page 2 (read link from links)
relative_tags_to_desired_linksp2= soup.find_all("a","blog-pager-older-link") 
for tagp2 in relative_tags_to_desired_linksp2:
    desired_elementp2 = tagp2.get("href")

    # read link from elementp2
    htmltextp2 = urllib2.urlopen(desired_elementp2).read()
    soupp2 = BeautifulSoup(htmltextp2)

    # extract link in elementp2
    relative_tags_to_desired_linksp2 = soupp2.find_all("h3", "post-title entry-title") 
    for taggingp2 in relative_tags_to_desired_linksp2:
        desired_elementp22 = taggingp2.next_element.next_element
        print desired_elementp22.get("href")

# page 3 (read link from desired_elementp2)
relative_tags_to_desired_linksp3 = soupp2.find_all("a","blog-pager-older-link") 
for tagp3 in relative_tags_to_desired_linksp3:
    desired_elementp3 = tagp3.get("href")

     # read link from desired_elementp3
    htmltextp3 = urllib2.urlopen(desired_elementp3).read()
    soupp3 = BeautifulSoup(htmltextp3)

    #extract link in desired_elementp3
    relative_tags_to_desired_linksp32 = soupp3.find_all("h3", "post-title entry-title") 
    for taggingp3 in relative_tags_to_desired_linksp32:
        desired_elementp32 = taggingp3.next_element.next_element
        print desired_elementp32.get("href")

Link in link.txt file:

http://ift.tt/1GTAa5L

Thank you.

Hierarchical clustering a pairwise distance matrix of precomputed distances

I have a pairwise distance dataframe that I've made with pandas:

#Get files
import glob
import itertools
one_dimension = glob.glob('*.pdb')

dataframe = []
for combo in itertools.combinations(one_dimension,2):
    pdb_1 = combo[0]
    pdb_2 = combo[1]
    entry = { 'pdb_1' : pdb_1, 'pdb_2', 'rmsd': get_rmsd(pdb_1,pdb_2)
    dataframe.append(entry)

import pandas
dataframe = Dataframe(dataframe)
dataframe

All I want to do is cluster the dataframe in such a way where all clusters contain pdbs that are less than some cutoff ( lets say less than 2). I have read that complete linkage is the way to go.

For instance:

1. pdb_1,pdb_2 have an rmsd 1.56
2. pdb_3,pdb_2 have an rmsd 1.03
3. pdb_2, pdb_1 have an rmsd of 1.60

So they are can all appear in a cluster together. But if any new pdb tries to be added to the cluster, if it is > 2 for any member already in the cluster, it will be rejected.

I understand that this is a complete linkage with a cutoff.

I have looked into scipy.cluster.hierarchy.linkage, but I'm having an extremely hard time formatting the array to enter into the linkage.

• What is the best way to complete this task?

• How do I go from my dataframe to something that can be useable by
  scipy.cluster?

• Should I turn it into an R dataframe?

• How do I find out which members are in the cluster if I transform the pairwise distance to an array.

I have found this, this, and this question similar, and found this tutorial

How to make a py.test failure trigger outside functions?

I am currently writing a script that installs my software-under-test then automatically runs my smoke tests using py.test. If a failure occurs during any of these tests, I would like to tell my software to not publish the software to the build servers. This is basically how it goes in pseudo-code:

def install_build_and_test():
    # some python code installs some_build
    install_my_build(some_build)

    # then I want to test my build
    subprocess.Popen(["py.test", "smoke_test_suite.py"])

    # If any failures occurred during testing, do not publish build 
    if test_failures is True:
        print "Build will not publish because there were errors in your logs"

    if test_failures is False:
        publish_build(some_build)

My question here is how do I use pytest failures to flag my install_and_test_build code to not publish some_build?

Django - Time since model created

How du you calculate the time since a model was created? The model has the following field:

created_at = models.DateTimeField(auto_now_add=True)

This is what I tried:

from datetime import datetime, timedelta

@property
def time_since_created(self):
    return (datetime.now()-self.created_at).total_seconds()

What I do not understand is that it crashes without giving me any error messages. If I wrap it in a try/except block like this:

@property
    def time_since_created(self):
        try:
            return (datetime.now()-self.created_at).total_seconds()
        except Exception as e:
            try:
                print("Error:" + e)
            except:
                print("Error: An exception occured when trying to print exception")

The error I will get is "Error: An exception occured when trying to print exception". If I do not include the last try/catch block I won't get any output at all.

Do anyone have any ideas?

Vectorize iterative addition in python arrays

For each element in a randomized array of 2D indices (with potential duplicates), I want to "+=1" to the corresponding grid in a 2D zero array. However, I don't know how to optimize the computation. Using the standard for loop, as shown here,

def interadd():
    U = 100 
    input = np.random.random(size=(5000,2)) * U
    idx = np.floor(input).astype(np.int) 

    grids = np.zeros((U,U))      
    for i in range(len(input)):
        grids[idx[i,0],idx[i,1]] += 1
    return grids

the runtime can quite significant:

>> timeit(interadd, number=5000)
43.69953393936157

Is there a way to vectorize this iterative process?

Extract list from given list and given indices

I have a list containinig, more or less, random values. The list always has a fixed length. I have another list containing integer values. These values are always smaller than the length of the first list.

I want to calculate a list containing all values from the first list whose indices are described by the values in the second list. I came up with the following:

>>> values = ['000', '111', '222', '333', '444', '555', '666', '777']
>>> indices = [2, 4, 7]
>>> [v for i, v in enumerate(values) if i in indices]
['222', '444', '777']

As my lists are rather small (24 elements) this is OK for me. Anyway, I wonder if there is some more elegant solution which does not calculate a temporary list (with enumerate()).

pxssh Breaks When ssh Password Is Incorrect

I Am Trying To Connect to an ssh server in python using pxssh lib

#!/usr/bin/python

import pxssh

attempt = pxssh.pxssh()

if not attempt.login(IP, USERNAME, PASSWORD):

    print("SSH LOGIN FAILED USING USERNAME : " + USERNAME + " AND PASSWORD : " + PASSWORD)

else:

    print("LOGIN WAS SUCCESSFUL USING USERNAME : " + USERNAME + " AND PASSWORD : " + PASSWORD)

    attempt.sendline("pwd")

    attempt.prompt()

    print attempt.before

break

it works fine when password is correct and logs into server and executes

pwd

but when password is incorrect or login failes for any reason the program breaks i expect this :

print("SSH LOGIN FAILED USING USERNAME : " + USERNAME + " AND PASSWORD : " + PASSWORD)

but instead i get this error:

  File "./ssh.py", line 19, in <module>
    if not attempt.login(IP, USERNAME, PASSWORD):
  File "/usr/lib/python2.7/dist-packages/pexpect/pxssh.py", line 292, in login
    raise ExceptionPxssh ('password refused')
pexpect.pxssh.ExceptionPxssh: password refused

Import-Module virtualenvwrapper

I am using powershell. I successfully installed virtualenvwrapper-powershell.

But when I typed 'Import-Module virtualenvwrapper', I got an error as following:

PS C:\Python27> Import-Module virtualenvwrapper Import-Module : The specified module 'virtualenvwrapper' was not loaded because no valid module file was found in any module directory. At line:1 char:1

Import-Module virtualenvwrapper ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ CategoryInfo : ResourceUnavailable: (virtualenvwrapper:String) [Import-Module], FileNotFoundException FullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand

It would be great if you could give me some advice...

Python regex: Matching a URL

I have some confusion regarding the pattern matching in the following expression. I tried to look up online but couldn't find an understandable solution:

imgurUrlPattern = re.compile(r'(http://i.imgur.com/(.*))(\?.*)?')

What exactly are the parentheses doing ? I understood up until the first asterisk , but I can't figure out what is happening after that.

Repeat print after every n prints

Given a variable n.

Now I want to print "Yes" n times then "No" n times

and then repeat the entire thing over and over.

How to do this in Python in shortest possible manner. I am looking for something succinct.

python spur ssh - use set command

I am using the below commands to ssh into a windows server and I am trying to use the set command. This is what I am using locally which works:

set ASPERA_SCP_PASS=myPassword
ascp /directory/Test4.mov myAccount@10.1.1.1:/TEST

And here is my python commands:

shell = spur.SshShell(hostname=10.0.0.1, username=Wusername, password=Wpassword, missing_host_key=spur.ssh.MissingHostKey.accept)
result = shell.run(["set", "ASPERA_SCP_PASS=myPassword])
result = shell.run(["ascp.exe", "/directory/Test4.mov", "myAccount@10.1.1.1:/TEST"])

When running the above I am successfully connecting via ssh, but I am getting this error on the set command:

spur.results.RunProcessError: return code: 127
output: b''
stderr output: b'sh: line 0: exec: set: not found\n'

error when using scrapy startproject in python2.7

I successfully installed Scrapy under python2.7,and when i created a new project using scrapy startproject example,an exception raised:

Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.4/site-packages/pkg_resources.py", line 2697, in <module>
    working_set.require(__requires__)
  File "/usr/local/lib/python3.4/site-packages/pkg_resources.py", line 669, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.4/site-packages/pkg_resources.py", line 572, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: six>=1.5.2

any ideas?thanks in advance!

how to pass value from output to a string in python

I've been trying to make an application in python and I'm new to python. Well, what I actually want to do is that . I want the feedparser to read the values from an RSS of a website... say reddit... and then I want to make that output as a stringand pass the value further to my code... my code right now..

import feedparser
import webbrowser

feed = feedparser.parse('http://ift.tt/11qcD9m')
print feed['entries'][1]['title'] 
print feed['entries'][1]['link'] 

It is working right now.. it parses the feed and I get the output I want... Now, I want to use the "link" from the "print feed['entries'][1]['link'] " and use it in the code further... how can I do so..? To be more specific.. I want to open that URL in my browser... I concluded to something like this..

import feedparser
import webbrowser

feed = feedparser.parse('http://ift.tt/11qcD9m')
print feed['entries'][1]['title'] 
print feed['entries'][1]['link'] 

mystring = 'feed['entries'][1]['link']'
webbrowser.open('mystring')

It is of course not working... Please Help... if you need to know anything else.. please let me know...

How to place a value outside stacked bar chart in xlsxwriter

in a stacked bar chart, is there's a way to put the value outside the chart using xlsxwriter?

Server side: Fill form on a webpage

Lets say there's a form on a webpage with URL xyz.com . The form has some text input fields, some drop down selections to be made etc. I need to write a server side script to visit this web page and fill and submit the form. Note that the server doesn't have a browser environment.

What should I do to achieve this? Any techniques in PHP (using cURL) or Python or echoing Java script via PHP ? I'd also have the form field IDs with me.

Puthon 2.7: Comparison method, compairing 2 points returning boolean value [duplicate]

This question already has an answer here:

• Elegant ways to support equivalence (“equality”) in Python classes 8 answers

I am using an equality method to compare the x and y coordinates of a point to another point, returning a boolean value. If the points have the same x and y coordinates, I want to return True, otherwise, I want to return False.

class Point(Geometry):
    def __init__(self,x,y):
        Geometry.__init__(self,id=0)
        self.x = x
        self.y = y

    def __str__(self):
        return'x = %.2f,y = %.2f'%(int(self.x*100)/100.0,int(self.y*100)/100.0)


    def __eq__(self,p1,p2):
        if p1 == p2: #boolean method?
            return True
        else:
            return False

Output:

>>>p1 = Point(-2,1)
>>> p2 = Point(1,5)
>>> print p1
x = -2.00,y = 1.00
>>> print p2
x = 1.00,y = 5.00
>>> p1.equal(p2)

Error:

p1.equal(p2) AttributeError: 'Point' object has no attribute 'equal'

I was expecting a True or False but I must not be using the comparison method properly. Should there b another if statement in place of the if not condition? Such as:

if p1!=p2:
    return False

Displaying Image (in bytes) in PyQt

I need to know how to display an image in bytes in GUI. I am taking an image from google static map API with .content I get the image in bytes like this:

import requests

a = requests.get('http://ift.tt/1IkZW5p')
print(a.content)

I want to display it in the interface i am creating. I know that i can save the bytes in an image and then create a QPixmap loading the image and adding it to a scene or maybe a Qlabel, but can I display the image in the interface without doing this?

I would appreciate any help.

predict classes of test data using k folding using sklearn

I am working on a data mining project and I am using the sklearn package in python for classifying my data.

in order to train my data and evaluate the quality of the predicted values, I am using the sklearn.cross_validation.cross_val_predict function.

however, when I try to run my model on the test data, it asks for the base class, which are not available.

I have seen (possible) work-arounds using the sklearn.grid_search.GridSearchCV function but am loathe to use such a method for a fixed set of parameters.

going throught the sklearn.cross_validation documentation, I have come across the cross_val_score function. Since I am fairly new to the world of classification problems I am not quite sure if this the function which would solve my problem.

Any help will be awesome!

Thanks!

edit:

Hello! I get the impression I was fairly vague with my original query. I'll try to detail what it is that I am exactly doing. Here goes:

I have generated 3 numpy.ndarrays X,X_test and y with nrows = 10158, 22513 and 10158 respectively which correspond to my train data, test data and class labels for the train data.

Thereafter, I run the following code :

    from sklearn.svm import SVC
    from sklearn.cross_validation import cross_val_predict
    clf = SVC()
    testPred = cross_val_predict(clf,X,y,cv=2)

This works fine and I can then use stemPred and y as mentioned in the tutorials.

However, I am looking to predict the classes of X_test. The error message is rather self-explanatory and says:

    ValueError: Found arrays with inconsistent numbers of samples: [10158 22513]

The current work around (I do not know if this is a work around or the only way to do it) I am using is:

    from sklearn import grid_search
    # thereafter I create the parameter grid (grid) and appropriate scoring function (scorer)
    model = grid_search.GridSearchCV(estimator = clf, param_grid = grid, scoring = scorer, refit = True, cv = 2, n_jobs = -1)
    model.fit(X,y)
    model.best_estimator_.fit(X,y)
    testPred = model.best_estimator_.predict(X_test)

This technique works fine for the time-being; however, if I didn't have to use the GridSearchCV function I'd be able to sleep much better.

Run .py when .txt is opened?

This feels like a simple question, but I don't know if there's a simple answer. I basically just need to make a .py file run when I open up a .txt. I don't really want a loop to check and then run it, preferably i'd like to bundle the txt and the py so all the user sees is a .txt file. Thanks!

AES decryption in python with PyCrypto

I'm trying to write an encryption/decryption program for Python, using PyCrypto The way the program works is that you specify an input/output, choose your input mode (for the sake of this question AES), and the program encrypts the input to the output file while also writing the encryption key to a file called encryption.key. To decrypt, you specify an input/output again, choose your mode (AES) again, and then the program loads the key from the encryption.key and uses it to decrypt the input.

The problem is, I'm a novice at python and cryptography and don't know how to fix the problem I'm getting. The problem is this: encryption works fine (as far as I can tell), but on decryption binascii throws an error about the keyfile having invalid padding. I'm not sure how to pad correctly- I tried implementing some code to pad it (based off the message pad) but it still throws the error. Does anyone know how to pad this key correctly?

Forgive me for being such a noob, everything I know about AES encryption came from me reverse-engineering some code I found in the source of Google's Keyczar's python implementation.

Now that I think about it, I don't really know why I just didn't use keyczar...

CODE:

#!/usr/bin/python
# -*- coding: utf-8 -*-
# P-Cypher-Dev
# Made in 2015 by Mateo Guynn
# v0.05 dev
# Using AES encryption \ Caesar Cipher
# DEV VERSION: Possibly unstable, contains better code.
# Changelog:
"""
v0.02d
- Improved Caesar Cipher
- Added binary Cipher converter (fail)
-------------FILE BROKEN------------
"""
"""
v0.03d
- Added ability to create new output files
- Fixed code not quitting on abort
- DEL : binary Cipher converter 
---------------STABLE---------------

"""
"""
v0.04d
- DEL : Caesar Cypher
- Added 16/32/64-byte AES encryption
- Binary and text now handled in same manner
-------------FILE BROKEN------------
(encryption works, decryption does not)

"""
"""
v0.05d
- Changed AES encryption to Google's way
- Fixed Key entry
- Added mode switcher 
- Readded Caesar Cipher to mode switcher (defaults to AES)
----------------O.K.----------------
       (AES decryption broken)

"""
"""
v0.05g (v0.05 gui)
- Initial tests of a GUI


"""
import os
import sys#!/usr/bin/python
# -*- coding: utf-8 -*-
# P-Cypher-Dev
# Made in 2015 by Mateo Guynn
# v0.05 dev
# Using AES encryption \ Caesar Cipher
# DEV VERSION: Possibly unstable, contains better code.
# Changelog:
"""
v0.02d
- Improved Caesar Cipher
- Added binary Cipher converter (fail)
-------------FILE BROKEN------------
"""
"""
v0.03d
- Added ability to create new output files
- Fixed code not quitting on abort
- DEL : binary Cipher converter 
---------------STABLE---------------

"""
"""
v0.04d
- DEL : Caesar Cypher
- Added 16/32/64-byte AES encryption
- Binary and text now handled in same manner
-------------FILE BROKEN------------
(encryption works, decryption does not)

"""
"""
v0.05d
- Changed AES encryption to Google's way
- Fixed Key entry
- Added mode switcher 
- Readded Caesar Cipher to mode switcher (defaults to AES)
----------------O.K.----------------
       (AES decryption broken)

"""
"""
v0.05g (v0.05 gui)
- Initial tests of a GUI


"""
import os
import sys
from Crypto.Cipher import AES
from Crypto import Random
from Crypto.Util import randpool
import base64

currentEncryptions = ['AES', 'Caesar Cipher']

def getMode():

    while True:

        eOrD = input('\nDo you wish to encrypt or decrypt a message? ')

        mode = eOrD.lower()

        if mode in 'encrypt e decrypt d'.split():

            return mode

        else:

            sys.exit('\nEnter either "encrypt" or "e" or "decrypt" or "d". Capital letters are allowed.\n')



def getMessage():


    inputFile = open(input('\nPlease enter the name of the file you want to encrypt/decrypt. You may use relative or full paths. \nPlease, remember the file extension(s)! ')).read()
    try:
        print ('\nThe contents of the file are: \n%s\n' % inputFile)
        return inputFile
    except IOError as e:
        sys.exit('Unable to open file (the file does not exist or P-Cypher does not have permission to view it).\n Aborting.')
    except FileNotFoundError as e:
        sys.exit('Unable to open file (the file does not exist or P-Cypher does not have permission to view it).\n Aborting.')


def getAESCipher(mode, message):
    block_size = 16 # For AES, this is the only working value
    key_size = 32 # Size of crypto key (possibly changes in getKey())
    aesmode = AES.MODE_CBC # More secure mode
    if mode[0] == 'e':
        key_bytes = randpool.RandomPool(512).get_bytes(key_size)
        pad_key = block_size - len(key_bytes) % block_size
        key_bytes_write = key_bytes + pad_key * bytearray(pad_key)
        open('decryption.key', 'wb+').write(key_bytes_write)
        print('\nYour keyfile is: decryption.key\n')
        pad = block_size - len(message) % block_size
        data = message + pad * chr(pad)
        iv_bytes = randpool.RandomPool(512).get_bytes(block_size)
        encrypted_bytes = iv_bytes + AES.new(key_bytes,aesmode,iv_bytes).encrypt(data)
        encrypted = base64.urlsafe_b64encode(encrypted_bytes)
        return encrypted
    else: 
        decryptb = base64.urlsafe_b64decode(message)
        decrypted_ivbytes = decryptb[:block_size]
        decrypt = decryptb[block_size:]
        print('\nAuto-searching for decryption.key...')
        try:
            key_bytes = base64.urlsafe_b64decode(open('decryption.key', 'rb').read())
        except IOError as io:
            key_bytes = base64.urlsafe_b64decode(open(input('decryption.key not found. If you have an alternate keyfile, please enter its name now. ')), 'rb').read
        except FileNotFoundError as fnf:
            key_bytes = base64.urlsafe_b64decode(open(input('decryption.key not found. If you have an alternate keyfile, please enter its name now. '), 'rb').read())
        decrypted = AES.new(key_bytes, aesmode, decrypted_ivbytes).decrypt(decryptb)
        pad = ord(decrypted[-1])
        decrypted = decrypted[:-pad]
        return decrypted

def getOutput():


    outputFile = input('\nPlease specify an output file. \nDon\'t forget the file extension! ')

    outputCheck = input('\nYour message will be encrypted/decrypted into the following output file: %s\n\nIs this okay? (y/n) ' % outputFile).lower()


    if outputCheck in 'y yes yeah ok'.split():

        try:
            if os.stat(outputFile).st_size != 0:
                overwriteFile = input('The file you specified is not empty. Do you want to overwrite it? (y/n)')
                if overwriteFile in 'y_yes_yeah_yes please_please_ok'.split('_'):
                    return outputFile
                else:
                    sys.exit('Aborting.') 
            else:
                return outputFile
        except IOError as ioerror:
            createNewFile = input('The file you specified does not exist. Shall I create one? (y/n) ').lower()
            if createNewFile in 'y_yes_yeah_yes please_ok_please'.split('_'):
                oF = open(outputFile, 'w+')
                oF.close()
                return outputFile

            else:
                sys.exit('Aborting...')
    elif outputCheck in 'n no'.split():
        sys.exit('\nAborting...\n')
    else:
        sys.exit('\nAborting.\n')

def getEncrypt(mode,message,cipherMode):
    case = cipherMode

    if case == 0: # If the cipherMode is set to AES -- implemented

        return getAESCipher(mode,message)


    elif case == 1: # If the cipherMode is set to Blowfish -- not implemented

        return getBlowfishCipher(mode,message)


    elif case == 2: # If the cipherMode is set to SHA -- not implemented

        return getSHAHash(mode,message)


    elif case == 3: # If the cipherMode is set to HMAC -- not implemented

        return getHMACHash(mode,message)


    elif case == 4: # If the cipherMode is set to PublicKey -- not implemented

        return getPublicKey(mode,message)


    elif case == 5: # If the cipherMode is set to Caesar -- implemented

        return getCaesarCipher(mode,message)


    else: # If no case is found

        print('No cipher mode found. Defaulting to AES.')
        return getAESCipher(mode,message)



def getShiftNum(mode): # Used in the Caesar Cipher to determine the amount of characters to shift the message.

    if mode[0] == 'e':
        shiftNumInput = input('\nHow many characters would you like to shift the message? (1-26) ')
    else:
        shiftNumInput = input('\nPlease enter your shift number. (1-26) ')
    try:
        shiftNum = int(shiftNumInput)

        if shiftNum >= 1 and shiftNum <= 26:

            return shiftNum

        else:

            sys.exit('\nSorry, but that\'s not in the specified range. Aborting.\n')

    except TypeError:

        sys.exit('\nSorry, but that\'s not an integer. Aborting.\n')

    except:

        sys.exit('Input error. Aborting.')

def getBlowfishCipher(mode,message):

    # This ain't implemented yet! Coming soon (possibly)
    sys.exit('Sorry, but blowfish encryption is not currently implemented. Available encryptions: %s' % currentEncryptions)

def getSHAHash(mode,message):

    # This ain't implemented yet! Coming soon (possibly)
    sys.exit('Sorry, but SHA hash encryption is not currently implemented. Available encryptions: %s' % currentEncryptions)

def getHMACHash(mode,message):

    # This ain't implemented yet! Coming soon (possibly)
    sys.exit('Sorry, but HMAC hash encryption is not currently implemented. Available encryptions: %s' % currentEncryptions)

def getPublicKey(mode,message):

    # This ain't implemented yet! Coming soon (possibly)
    sys.exit('Sorry, but public keys are not currently implemented. Available encryptions: %s' % currentEncryptions)

def getCaesarCipher(mode,message):

    key = getShiftNum(mode)

    encryptedMessage = ''

    if mode[0] == 'd':

        key = -key

    for symbol in message:

        if symbol.isalpha():

            num = ord(symbol)
            num += key

            if symbol.isupper():
                if num > ord('Z'):
                    num -= 26
                elif num < ord('A'):
                    num += 26
            elif symbol.islower():
                if num > ord('z'):
                    num -= 26
                elif num < ord('a'):
                    num += 26

            encryptedMessage += chr(num)
        else:
            encryptedMessage += symbol
    return encryptedMessage

def getCipherMode():
    cipherModeInput = input('\nPlease select your encryption mode. Available encryptions are: %s. ' % currentEncryptions).lower()

    if cipherModeInput in 'aes_advanced encryption standard_a'.split('_'):

        print('\nEncrypting with AES.')
        cipherMode = 0
        return cipherMode

    elif cipherModeInput in 'blowfish b blow fish'.split():

        print('\nEncrypting with Blowfish.')
        cipherMode = 1
        return cipherMode

    elif cipherModeInput in 'sha_sha-256_sha-512_sha-1_sha-2_sha-3_secure hash algorithm_secure hash_sha 256_ sha 512_sha 1_sha 2_sha 3'.split('_'):

        print('\nEncrypting with SHA.')
        cipherMode = 2
        return cipherMode

    elif cipherModeInput in 'hmac_hash-based message authentication code_hash based message authentication code_h'.split('_'):

        print('\nEncrypting with HMAC.')
        cipherMode = 3
        return cipherMode

    elif cipherModeInput in 'p_pk_k_public_key_public key'.split('_'):

        print('\nEncrypting with a public key.')
        cipherMode = 4
        return cipherMode

    elif cipherModeInput in 'caesar_cipher_cypher_caesar cipher_caesar cypher_shifter'.split('_'):

        print('\nEncrypting with a Caesar Cipher.')
        cipherMode = 5
        return cipherMode

    else:

        cipherMode = 20
        return cipherMode

def startup():
    print("\nP-Cypher Alpha starting up...\n\nv0.05 dev\nMateo Guynn\n2015\n")
    mode = getMode()

    message = getMessage()

    cipherMode = getCipherMode()
    if cipherMode != 5:
        try:
            open(getOutput(), 'wb').write(getEncrypt(mode,message,cipherMode))
        except IOError:
            sys.exit('Oh noes! Something has gone terribly wrong!')
        except FileNotFoundError:
            sys.exit('Your input file was not found.')
    else:
        try:
            open(getOutput(), 'w+').write(getEncrypt(mode,message,cipherMode))
        except IOError:
            sys.exit('Oh noes! Something has gone terribly wrong!')
        except FileNotFoundError:
            sys.exit('Your input file was not found.')

    print('\nDone.')

startup()
from Crypto.Cipher import AES
from Crypto import Random
from Crypto.Util import randpool
import base64

currentEncryptions = ['AES', 'Caesar Cipher']

def getMode():

    while True:

        eOrD = input('\nDo you wish to encrypt or decrypt a message? ')

        mode = eOrD.lower()

        if mode in 'encrypt e decrypt d'.split():

            return mode

        else:

            sys.exit('\nEnter either "encrypt" or "e" or "decrypt" or "d". Capital letters are allowed.\n')



def getMessage():


    inputFile = open(input('\nPlease enter the name of the file you want to encrypt/decrypt. You may use relative or full paths. \nPlease, remember the file extension(s)! ')).read()
    try:
        print ('\nThe contents of the file are: \n%s\n' % inputFile)
        return inputFile
    except IOError as e:
        sys.exit('Unable to open file (the file does not exist or P-Cypher does not have permission to view it).\n Aborting.')
    except FileNotFoundError as e:
        sys.exit('Unable to open file (the file does not exist or P-Cypher does not have permission to view it).\n Aborting.')


def getAESCipher(mode, message):
    block_size = 16 # For AES, this is the only working value
    key_size = 32 # Size of crypto key (possibly changes in getKey())
    aesmode = AES.MODE_CBC # More secure mode
    if mode[0] == 'e':
        key_bytes = randpool.RandomPool(512).get_bytes(key_size)
        pad_key = block_size - len(key_bytes) % block_size
        key_bytes_write = key_bytes + pad_key * bytearray(pad_key)
        open('decryption.key', 'wb+').write(key_bytes_write)
        print('\nYour keyfile is: decryption.key\n')
        pad = block_size - len(message) % block_size
        data = message + pad * chr(pad)
        iv_bytes = randpool.RandomPool(512).get_bytes(block_size)
        encrypted_bytes = iv_bytes + AES.new(key_bytes,aesmode,iv_bytes).encrypt(data)
        encrypted = base64.urlsafe_b64encode(encrypted_bytes)
        return encrypted
    else: 
        decryptb = base64.urlsafe_b64decode(message)
        decrypted_ivbytes = decryptb[:block_size]
        decrypt = decryptb[block_size:]
        print('\nAuto-searching for decryption.key...')
        try:
            key_bytes = base64.urlsafe_b64decode(open('decryption.key', 'rb').read())
        except IOError as io:
            key_bytes = base64.urlsafe_b64decode(open(input('decryption.key not found. If you have an alternate keyfile, please enter its name now. ')), 'rb').read
        except FileNotFoundError as fnf:
            key_bytes = base64.urlsafe_b64decode(open(input('decryption.key not found. If you have an alternate keyfile, please enter its name now. '), 'rb').read())
        decrypted = AES.new(key_bytes, aesmode, decrypted_ivbytes).decrypt(decryptb)
        pad = ord(decrypted[-1])
        decrypted = decrypted[:-pad]
        return decrypted

def getOutput():


    outputFile = input('\nPlease specify an output file. \nDon\'t forget the file extension! ')

    outputCheck = input('\nYour message will be encrypted/decrypted into the following output file: %s\n\nIs this okay? (y/n) ' % outputFile).lower()


    if outputCheck in 'y yes yeah ok'.split():

        try:
            if os.stat(outputFile).st_size != 0:
                overwriteFile = input('The file you specified is not empty. Do you want to overwrite it? (y/n)')
                if overwriteFile in 'y_yes_yeah_yes please_please_ok'.split('_'):
                    return outputFile
                else:
                    sys.exit('Aborting.') 
            else:
                return outputFile
        except IOError as ioerror:
            createNewFile = input('The file you specified does not exist. Shall I create one? (y/n) ').lower()
            if createNewFile in 'y_yes_yeah_yes please_ok_please'.split('_'):
                oF = open(outputFile, 'w+')
                oF.close()
                return outputFile

            else:
                sys.exit('Aborting...')
    elif outputCheck in 'n no'.split():
        sys.exit('\nAborting...\n')
    else:
        sys.exit('\nAborting.\n')

def getEncrypt(mode,message,cipherMode):
    case = cipherMode

    if case == 0: # If the cipherMode is set to AES -- implemented

        return getAESCipher(mode,message)


    elif case == 1: # If the cipherMode is set to Blowfish -- not implemented

        return getBlowfishCipher(mode,message)


    elif case == 2: # If the cipherMode is set to SHA -- not implemented

        return getSHAHash(mode,message)


    elif case == 3: # If the cipherMode is set to HMAC -- not implemented

        return getHMACHash(mode,message)


    elif case == 4: # If the cipherMode is set to PublicKey -- not implemented

        return getPublicKey(mode,message)


    elif case == 5: # If the cipherMode is set to Caesar -- implemented

        return getCaesarCipher(mode,message)


    else: # If no case is found

        print('No cipher mode found. Defaulting to AES.')
        return getAESCipher(mode,message)



def getShiftNum(mode): # Used in the Caesar Cipher to determine the amount of characters to shift the message.

    if mode[0] == 'e':
        shiftNumInput = input('\nHow many characters would you like to shift the message? (1-26) ')
    else:
        shiftNumInput = input('\nPlease enter your shift number. (1-26) ')
    try:
        shiftNum = int(shiftNumInput)

        if shiftNum >= 1 and shiftNum <= 26:

            return shiftNum

        else:

            sys.exit('\nSorry, but that\'s not in the specified range. Aborting.\n')

    except TypeError:

        sys.exit('\nSorry, but that\'s not an integer. Aborting.\n')

    except:

        sys.exit('Input error. Aborting.')

def getBlowfishCipher(mode,message):

    # This ain't implemented yet! Coming soon (possibly)
    sys.exit('Sorry, but blowfish encryption is not currently implemented. Available encryptions: %s' % currentEncryptions)

def getSHAHash(mode,message):

    # This ain't implemented yet! Coming soon (possibly)
    sys.exit('Sorry, but SHA hash encryption is not currently implemented. Available encryptions: %s' % currentEncryptions)

def getHMACHash(mode,message):

    # This ain't implemented yet! Coming soon (possibly)
    sys.exit('Sorry, but HMAC hash encryption is not currently implemented. Available encryptions: %s' % currentEncryptions)

def getPublicKey(mode,message):

    # This ain't implemented yet! Coming soon (possibly)
    sys.exit('Sorry, but public keys are not currently implemented. Available encryptions: %s' % currentEncryptions)

def getCaesarCipher(mode,message):

    key = getShiftNum(mode)

    encryptedMessage = ''

    if mode[0] == 'd':

        key = -key

    for symbol in message:

        if symbol.isalpha():

            num = ord(symbol)
            num += key

            if symbol.isupper():
                if num > ord('Z'):
                    num -= 26
                elif num < ord('A'):
                    num += 26
            elif symbol.islower():
                if num > ord('z'):
                    num -= 26
                elif num < ord('a'):
                    num += 26

            encryptedMessage += chr(num)
        else:
            encryptedMessage += symbol
    return encryptedMessage

def getCipherMode():
    cipherModeInput = input('\nPlease select your encryption mode. Available encryptions are: %s. ' % currentEncryptions).lower()

    if cipherModeInput in 'aes_advanced encryption standard_a'.split('_'):

        print('\nEncrypting with AES.')
        cipherMode = 0
        return cipherMode

    elif cipherModeInput in 'blowfish b blow fish'.split():

        print('\nEncrypting with Blowfish.')
        cipherMode = 1
        return cipherMode

    elif cipherModeInput in 'sha_sha-256_sha-512_sha-1_sha-2_sha-3_secure hash algorithm_secure hash_sha 256_ sha 512_sha 1_sha 2_sha 3'.split('_'):

        print('\nEncrypting with SHA.')
        cipherMode = 2
        return cipherMode

    elif cipherModeInput in 'hmac_hash-based message authentication code_hash based message authentication code_h'.split('_'):

        print('\nEncrypting with HMAC.')
        cipherMode = 3
        return cipherMode

    elif cipherModeInput in 'p_pk_k_public_key_public key'.split('_'):

        print('\nEncrypting with a public key.')
        cipherMode = 4
        return cipherMode

    elif cipherModeInput in 'caesar_cipher_cypher_caesar cipher_caesar cypher_shifter'.split('_'):

        print('\nEncrypting with a Caesar Cipher.')
        cipherMode = 5
        return cipherMode

    else:

        cipherMode = 20
        return cipherMode

def startup():
    print("\nP-Cypher Alpha starting up...\n\nv0.05 dev\nMateo Guynn\n2015\n")
    mode = getMode()

    message = getMessage()

    cipherMode = getCipherMode()
    if cipherMode != 5:
        try:
            open(getOutput(), 'wb').write(getEncrypt(mode,message,cipherMode))
        except IOError:
            sys.exit('Oh noes! Something has gone terribly wrong!')
        except FileNotFoundError:
            sys.exit('Your input file was not found.')
    else:
        try:
            open(getOutput(), 'w+').write(getEncrypt(mode,message,cipherMode))
        except IOError:
            sys.exit('Oh noes! Something has gone terribly wrong!')
        except FileNotFoundError:
            sys.exit('Your input file was not found.')

    print('\nDone.')

startup()

Social Scoring system for PC

I am writing a simple game to teach Japanese and it would be good to have some way for students to compare their results as I think competition is a good motivator. So I was reviewing social scoring systems like Scoreloop and Openfeint, but they do not exist anymore.

Also, my game only makes sense to play with a real keyboard - so it is and always will be a PC game - which is a big problem, since the replacements for these defunct platforms are only for Android or iOS.

How do I achieve this functionality with a PC game?

I need a leaderboard and perhaps a list of achievements, nothing more.

Do I need to set up an SQL server and code everything from scratch? This sounds cumbersome and messy. Are there any more elegant alternatives?

Also, my code is in Python - if it makes any difference.

Efficiently calculate multiple object groupings

The situation I'm working on is I'm having to group cards into groups based power and card properties. Each group needs a certain combined power level from its cards and each card in a group need to have a required property or require a lack of a property in order for the group to report that it is ok. For example:

group1:
    required_property = prop1
    required_power_level = 4

group2:
    required_property = prop2
    required_power_level = 3

group3:
    required_property = not prop1
    required_power_level = 5

Some groups can define wild as their required property and will take any card not in any other group; these groups still have a power level that must be met.

Each card has a single, integer power level but can have zero, one or more properties. Obviously if a card is in one group it can not also be in another group.

card1:
    power = 2
    properties = prop1

card2:
    power = 2
    properties = prop1, prop2

card3:
    power = 5
    properties = None

card4:
    power = 3
    properties = prop1, prop2

Ultimately I need to distribute the cards to try get all groups satisfied, this includes any wild groups included. There can be an arbitrary number of groups (usually no more than 2 at a given time) and cards, and card's powers are unbounded positive integers (typically 9 or smaller though) and can have a max of 6 properties, or have no property at all.

The only thing I'm looking for at the end of the algorithm is if all groups can be satisfied or not, given a set of cards; the actual grouping of cards is irrelevant in the end.

Any ideas? I'm just looking for an algorithm, but if it helps, the implementation will be in ironpython.

Checking ALL links within links from a source HTML, Python

My code is to search a Link passed in the command prompt, get the HTML code for the webpage at the Link, search the HTML code for links on the webpage, and then repeat these steps for the links found. I hope that is clear.

It should print out any links that cause errors.

Some more needed info:

The max visits it can do is 100. If a website has an error, a None value is returned.

Python3 is what I am using

eg:

s = readwebpage(url)... # This line of code gets the HTML code for the link(url) passed in its argument.... if the link has an error, s = None.

The HTML code for that website has links that end in p2.html, p3.html, p4.html, and p5.html on its webpage. My code reads all of these, but it does not visit these links individually to search for more links. If it did this, it should search through these links and find a link that ends in p10.html, and then it should report that the link ending with p10.html has errors. Obviously it doesn't do that at the moment, and it's giving me a hard time.

My code..

    url = args.url[0]
    url_list = [url]
    checkedURLs = []
    AmountVisited = 0
    while (url_list and AmountVisited<maxhits):
        url = url_list.pop()
        s = readwebpage(url)
        print("testing url: http",url)                  #Print the url being tested, this code is here only for testing..
        AmountVisited = AmountVisited + 1
        if s == None:
            print("* bad reference to http", url)
        else:
            urls_list = re.findall(r'href="http([\s:]?[^\'" >]+)', s) #Creates a list of all links in HTML code starting with...
            while urls_list:                                          #... http or https
                insert = urls_list.pop()            
                while(insert in checkedURLs and urls_list):
                    insert = urls_list.pop()
                url_list.append(insert)
                checkedURLs = insert 

Please help :)

Conversion of Python DateTime string into integer milliseconds

I would like to convert a UTC TimeDate stamp string into an integer value of milliseconds (might need to be a 64-bit quantity), so that it takes up less space when stored in a mySQL database column. This UTC string is being generated from another library, and I store it as a kind of per-user GUID.

Can datetime or dateutil convert this into a single integer value (like "milliseconds since epoch")? Or do I need to do that myself?

Parsing using this approach:

myDateTime = dateutil.parser.parse("2015-06-27T02:10:05.653000Z")
print("Parsed datetime String is {0}, ordinal value is {1}".format(myDateTime, myDateTime.toordinal()))

Gives the output:

Parsed datetime String is 2015-06-27 02:10:05.652999+00:00, ordinal value is 735776

…which only gives an ordinal value for the date. Further, if I have a time with an integer 653 milliseconds, then I want that parsed object to know it has 653 milliseconds, not 652999.

Having trouble determining a 2D feature matrix structure to feed into machine learning algorithm

I am training an emotion recognition system that detects emotions through facial movement as a result, I have formed a 4 dimensional matrix that I am trying to reduce to 2 dimensions.

Features that makes up the 4D matrix:
Number of videos (and each video will be assigned emotion label)
Number of frames per video
Direction of the facial landmarks per frame
Speed of the facial landmarks per frame

The important features that I am trying to train with:
The left side is the speed (hypotenuse between same facial landmark each frame)
The right side is direction (arctan of the x and y values of the same facial landmark each frame)

The 4D matrix that I am stuck with and trying to reduce to 2D

>> main.shape  
(60, 17, 68, 2)  
# 60 videos, 17 frames per video, 68 facial landmarks, 2 features (direction and speed)  
>> main  
array([[[[  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         ..., 
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ]],

        [[  1.        ,   1.        ],
         [  1.41421356,   0.78539816],
         [  1.41421356,   0.78539816],
         ..., 
         [  3.        ,   1.        ],
         [  3.        ,   1.        ],
         [  3.        ,   1.        ]],

        [[  0.        ,   0.        ],
         [ -1.41421356,   0.78539816],
         [ -1.41421356,   0.78539816],
         ..., 
         [  2.        ,   1.        ],
         [  3.        ,   1.        ],
         [  3.        ,   1.        ]],

        ..., 
        [[  1.        ,   1.        ],
         [  1.41421356,  -0.78539816],
         [  1.41421356,  -0.78539816],
         ..., 
         [ -1.41421356,   0.78539816],
         [  1.        ,   1.        ],
         [ -1.41421356,   0.78539816]],

        [[  2.23606798,  -0.46364761],
         [  2.82842712,  -0.78539816],
         [  2.23606798,  -0.46364761],
         ..., 
         [  1.        ,   0.        ],
         [  0.        ,   0.        ],
         [  1.        ,   1.        ]],

        [[ -1.41421356,  -0.78539816],
         [ -2.23606798,  -0.46364761],
         [ -2.23606798,  -0.46364761],
         ..., 
         [  1.41421356,  -0.78539816],
         [  1.41421356,  -0.78539816],
         [  2.23606798,  -1.10714872]]],


       [[[  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         ..., 
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ]],

        [[  2.        ,   1.        ],
         [  2.23606798,  -1.10714872],
         [  1.41421356,  -0.78539816],
         ..., 
         [ -2.        ,  -0.        ],
         [ -1.        ,  -0.        ],
         [ -1.41421356,  -0.78539816]],

        [[  2.        ,   1.        ],
         [ -2.23606798,   1.10714872],
         [ -1.41421356,   0.78539816],
         ..., 
         [  1.        ,   1.        ],
         [ -1.        ,  -0.        ],
         [ -1.        ,  -0.        ]],

        ..., 
        [[ -2.        ,  -0.        ],
         [ -3.        ,  -0.        ],
         [ -4.12310563,  -0.24497866],
         ..., 
         [  0.        ,   0.        ],
         [ -1.        ,  -0.        ],
         [ -2.23606798,   1.10714872]],

        [[ -2.23606798,   1.10714872],
         [ -1.41421356,   0.78539816],
         [ -2.23606798,   1.10714872],
         ..., 
         [ -2.23606798,   0.46364761],
         [ -1.41421356,   0.78539816],
         [ -1.41421356,   0.78539816]],

        [[  2.        ,   1.        ],
         [  1.41421356,   0.78539816],
         [  2.82842712,   0.78539816],
         ..., 
         [  1.        ,   1.        ],
         [  1.        ,   1.        ],
         [ -2.23606798,  -1.10714872]]],


       [[[  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         ..., 
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ]],

        [[  1.        ,   1.        ],
         [  0.        ,   0.        ],
         [  1.        ,   1.        ],
         ..., 
         [ -3.        ,  -0.        ],
         [ -2.        ,  -0.        ],
         [  0.        ,   0.        ]],

        [[  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         ..., 
         [  1.41421356,   0.78539816],
         [  1.        ,   0.        ],
         [  0.        ,   0.        ]],

        ..., 
        [[  1.        ,   0.        ],
         [  1.        ,   1.        ],
         [  0.        ,   0.        ],
         ..., 
         [  2.        ,   1.        ],
         [  3.        ,   1.        ],
         [  3.        ,   1.        ]],

        [[ -7.28010989,   1.29249667],
         [ -7.28010989,   1.29249667],
         [ -8.54400375,   1.21202566],
         ..., 
         [-22.02271555,   1.52537305],
         [ 22.09072203,  -1.48013644],
         [ 22.36067977,  -1.39094283]],

        [[  1.        ,   0.        ],
         [  1.41421356,  -0.78539816],
         [  1.        ,   0.        ],
         ..., 
         [ -1.41421356,  -0.78539816],
         [  1.        ,   1.        ],
         [  1.41421356,   0.78539816]]],


       ..., 
       [[[  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         ..., 
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ]],

        [[  5.38516481,   0.38050638],
         [  5.09901951,   0.19739556],
         [  4.47213595,  -0.46364761],
         ..., 
         [ -1.41421356,   0.78539816],
         [ -2.82842712,   0.78539816],
         [ -5.        ,   0.64350111]],

        [[ -6.32455532,   0.32175055],
         [ -6.08276253,  -0.16514868],
         [ -5.65685425,  -0.78539816],
         ..., 
         [  3.60555128,   0.98279372],
         [  5.        ,   0.92729522],
         [  5.65685425,   0.78539816]],

        ..., 
        [[ -3.16227766,  -0.32175055],
         [ -3.60555128,  -0.98279372],
         [  5.        ,   1.        ],
         ..., 
         [ 12.08304597,   1.14416883],
         [ 13.15294644,   1.418147  ],
         [ 14.31782106,   1.35970299]],

        [[  3.60555128,  -0.5880026 ],
         [  4.47213595,  -1.10714872],
         [  6.        ,   1.        ],
         ..., 
         [-20.39607805,   1.37340077],
         [-21.02379604,   1.52321322],
         [-22.09072203,   1.48013644]],

        [[  1.        ,   1.        ],
         [ -1.41421356,   0.78539816],
         [  1.        ,   1.        ],
         ..., 
         [  4.12310563,   1.32581766],
         [  4.        ,   1.        ],
         [  4.12310563,   1.32581766]]],


       [[[  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         ..., 
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ]],

        [[  0.        ,   0.        ],
         [  1.        ,   1.        ],
         [ -2.23606798,   1.10714872],
         ..., 
         [ -3.16227766,   0.32175055],
         [  1.        ,   1.        ],
         [  1.41421356,  -0.78539816]],

        [[  1.        ,   1.        ],
         [  1.        ,   1.        ],
         [  1.        ,   1.        ],
         ..., 
         [  3.        ,   1.        ],
         [  2.        ,   1.        ],
         [ -1.41421356,   0.78539816]],

        ..., 
        [[  5.38516481,  -1.19028995],
         [  4.47213595,  -1.10714872],
         [  4.12310563,  -1.32581766],
         ..., 
         [  2.23606798,  -0.46364761],
         [  1.        ,   1.        ],
         [ -1.        ,  -0.        ]],

        [[ -5.38516481,   1.19028995],
         [ -4.12310563,   1.32581766],
         [ -3.16227766,   1.24904577],
         ..., 
         [  0.        ,   0.        ],
         [  1.        ,   0.        ],
         [  1.41421356,  -0.78539816]],

        [[  8.06225775,   1.44644133],
         [ -7.07106781,  -1.42889927],
         [  6.        ,   1.        ],
         ..., 
         [ -3.16227766,  -0.32175055],
         [ -3.16227766,  -0.32175055],
         [ -3.16227766,  -0.32175055]]],


       [[[  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         ..., 
         [  0.        ,   0.        ],
         [  0.        ,   0.        ],
         [  0.        ,   0.        ]],

        [[ -2.23606798,   0.46364761],
         [ -1.41421356,   0.78539816],
         [ -2.23606798,   0.46364761],
         ..., 
         [  1.        ,   0.        ],
         [  1.        ,   0.        ],
         [  1.        ,   1.        ]],

        [[ -2.23606798,  -0.46364761],
         [ -1.41421356,  -0.78539816],
         [  2.        ,   1.        ],
         ..., 
         [  0.        ,   0.        ],
         [  1.        ,   0.        ],
         [  1.        ,   0.        ]],

        ..., 
        [[  1.        ,   0.        ],
         [  1.        ,   1.        ],
         [ -2.23606798,  -1.10714872],
         ..., 
         [ 19.02629759,   1.51821327],
         [ 19.        ,   1.        ],
         [-19.10497317,  -1.46591939]],

        [[  3.60555128,   0.98279372],
         [  3.60555128,   0.5880026 ],
         [  5.        ,   0.64350111],
         ..., 
         [  7.28010989,  -1.29249667],
         [  7.61577311,  -1.16590454],
         [  8.06225775,  -1.05165021]],

        [[ -7.28010989,   1.29249667],
         [ -5.        ,   0.92729522],
         [ -5.83095189,   0.5404195 ],
         ..., 
         [ 20.09975124,   1.47112767],
         [ 21.02379604,   1.52321322],
         [-20.22374842,  -1.42190638]]]])

The direction and speed features are quite valuable (the most important features) as it represents the movement of each facial landmark per frame and I am trying to get the machine learning algorithm to train base on that

I tried to reshape three of the dimensions into one long vector (just mushed speed, direction, and frame all together) and finally formed a 2D matrix, I fed it into sklearn SVM function and it produced a rather low accuracy. I expected this as I figured there is no way the ml algorithm would recognize that the difference between the features in the giant single matrix and assume that everything in the vector is the same features.

The 2D matrix I was forced to make to feed in sklearn SVM by forcing speed, direction, and video per frame all into one vector, and got a low accuracy with:

>> main
array([[  0.        ,   0.        ,   0.        , ...,  -0.78539816,
          2.23606798,  -1.10714872],
       [  0.        ,   0.        ,   0.        , ...,   1.        ,
         -2.23606798,  -1.10714872],
       [  0.        ,   0.        ,   0.        , ...,   1.        ,
          1.41421356,   0.78539816],
       ..., 
       [  0.        ,   0.        ,   0.        , ...,   1.        ,
          4.12310563,   1.32581766],
       [  0.        ,   0.        ,   0.        , ...,  -0.32175055,
         -3.16227766,  -0.32175055],
       [  0.        ,   0.        ,   0.        , ...,   1.52321322,
        -20.22374842,  -1.42190638]])
>> main.shape
(60, 2312)

I want to preserve the speed and direction features, but have to represent them in a 2D matrix that takes into account the frames in the video.

The emotion label will be attached to each of the 17 frames in each video. (so basically, the 17 frame video will be labeled as an emotion)

Is there any smart way in reshaping and reducing the 4D matrix that would accomplish this?

python regex use capture group to define another groups length { }

I am parsing hex data with python regex. I have the following packet structure:

'\xaa\x01\xFF\x44'

• \xaa - start of packet
• \x01 - data length [value can vary from 00-FF]
• \xFF - data
• \x44 - end of packet

i want to use python regex to indicate how much of the data portion of the packet to match as such:

r = re.compile('\xaa(?P<length>[\x00-\xFF]{1})(.*){?P<length>}\x44')

this compiles without errors, but it doesnt work (i suspect because it cannot convert the hex value to an appropriate integer) Is there a method by which this can be accomplished in python?

Background: I have been using erlang for packet unpacking and I was looking for something similar in python

how to get value from another class in python by hit button

I have some problem, i want get value from database in another class by hit button. I used methode string matching, when user input string and after taht user press button. System will matching what user input to database. what i want now is, i want the output in another form and absoluty in another class. I create some code like this.

class MyFrame(wx.Frame):
  def __init__(self, parent):
    wx.Frame.__init__(self, parent)
    wx.StaticText(self, -1, "Asking", pos=(20, 50))
    self.inpt = wx.TextCtrl(self, -1, pos=(10, 70), size=(250, 20), style= wx.EXPAND)
    wx.StaticText(self, -1, "Link", pos=(20, 135))
    self.opt = wx.ListBox(self, -1, pos=(10, 160), size=(480, 250), style= wx.TE_MULTILINE | wx.BORDER_SUNKEN)

    img = "search.png"
    image1 = wx.Image(img, wx.BITMAP_TYPE_ANY).ConvertToBitmap()
    self.btn1 = wx.BitmapButton(pan1, id=-1, bitmap=image1, pos=(85, 105), size = (image1.GetWidth()+5, image1.GetHeight()+5))

    self.btn1.Bind(wx.EVT_BUTTON, self.OnEnter, self.btn1)

    self.klik = self.opt.Bind(wx.EVT_LISTBOX_DCLICK, self.playVideo)

def playVideo(self, event):
    data = event.GetClientObject()
    link = data[0]
    ex = data[1]

    self.player.Load(link)
    self.pjs.Clear()
    self.pjs.AppendText(ex)

  def OnEnter(self, event):
    self.opt.SetLabel(self.PatMatch())

  def PatMatch(self):
    self.opt.Clear()
    con = lite.connect('media.db')  
    with con:
        cur = con.cursor()
        for row in cur.execute("select * from baru"):
            self.klmt = self.inpt.GetValue()
            if row[1] in self.klmt.lower():
                self.opt.Append(row[2], [row[4], row[3]])

class Kmp(wx.Frame):
  def __init__(self, parent, *args, **kwds):
      kwds["style"] = (wx.DEFAULT_FRAME_STYLE) & ~ (wx.RESIZE_BORDER|wx.RESIZE_BOX)
      wx.Frame.__init__(self,parent,title='Informasi', size=(500, 500), *args, **kwds)

      km2 = wx.StaticText(self, label='Word', pos=(20, 70))
      ttk2 = wx.StaticText(self, label=':', pos=(200,70))
      txt1 = wx.TextCtrl(self, -1, pos=(220, 70), size=(200, 30), style= wx.EXPAND)
      txt1.SetEditable(False)

      km3 = wx.StaticText(self, label='Total word', pos=(20, 110))
      ttk3 = wx.StaticText(self, label=':', pos=(200,110))
      txt2 = wx.TextCtrl(self, -1, pos=(220, 110), size=(200, 30), style= wx.EXPAND)
      txt2.SetEditable(False)

      km4 = wx.StaticText(self, label='POsisition Word', pos=(20,145))
      ttk4 = wx.StaticText(self, label=':', pos=(200,145))
      txt3 = wx.TextCtrl(self, -1, pos=(220, 150), size=(200, 30), style= wx.EXPAND)
      txt3.SetEditable(False)

      btn2 = wx.Button(self, -1, 'Exit', pos=(200, 200))

    def data(self):
        dt = MyFrame(self)
        klm = selt.inpt.GetValue()
        re.findall(r"[\w']+", klm)      

    def compute_prefix(word):
        word_length = len(word)
        prefix = [0] * word_length
        k = 0

        for q in xrange(1, word_length):
            while k > 0 and word[k] != word[q]:
                k = prefix[k - 1]

            if word[k + 1] == word[q]:
                k = k + 1
            prefix[q] = k
        return prefix

    def search(self, string, word):
        word_length = len(word)
        string_length = len(string)
        offsets = []
        if word_length > string_length:
            return offsets

        prefix = compute_prefix(word)
        q = 0
        for index, letter in enumerate(string):
            while q > 0 and word[q] != letter:
                q = prefix[q - 1]
            if word[q] == letter:
                q += 1
            if q == word_length:
                offsets.append(index - word_length + 1)
                q = prefix[q - 1]
        return offsets

    word = ""
    string = "" 

    def pro(self, event):
        pp = MyFrame(self)
        self.opt.SetLabel(self.PatMatch())

my database:

id  word        name_link   explain     link
--- ------      ----------- ---------   --------
1   python      Video1      test        C:\Users\Ihsan\Downloads\Video1.MP4
2   python      Video2      test1       C:\Users\Ihsan\Downloads\Video2.MP4
3   python      Video3      test2       C:\Users\Ihsan\Downloads\Video3.MP4

Python - print objects with long strings

Currently working with Scrapy and I'd like to see what I'm doing. But it seems as if Scrapy or probably Python cuts off longer strings if I want to print an object:

    products = Selector(response).xpath('//div[@class="s-item-container"]')
    pprint ( products )

Now I receive:

<Selector xpath='//div[@class="s-item-container"]' data=u'<div class="s-item-container"><div class'>,
<Selector xpath='//div[@class="s-item-container"]' data=u'<div class="s-item-container"><div class'>,
<Selector xpath='//div[@class="s-item-container"]' data=u'<div class="s-item-container"><div class'>,

in my shell - how could I print everything that products contains? (the HTML content is chopped off).

scrapy multiple start urls

i need to scrape this website for data, it has 10 categories with 180 pages each, so far, i managed to scrape the pages of a category like this: start_urls = ["http://ift.tt/1TW6ITG" % d for d in range(0, 180)] and it works, its scraping the 180 pages but i need to find a way to pass category names too.

I tried

start_urls = [
"http://ift.tt/1TW6ITG" % d for d in range(0, 180),
"http://ift.tt/1TW6HPs" % d for d in range(0, 180),
"http://ift.tt/1TW6HPu" % d for d in range(0, 180)
]

but it doesn't work (python error)

Any ideas?

RECAPTCHA_PUBLIC_KEY config not set with Flask-WTForms

I've gone over and over the documentation and keep getting 'RuntimeError: RECAPTCHA_PUBLIC_KEY config not set' when the Contact page attempts to load using Recaptcha with wtforms in Flask. I'm open to any kind of suggestions. I am running Flask on Ubuntu 14.04 using Google's ReCaptcha 2, FWIW.

EDIT: This issue has been resolved with the advice from dirn but I am now getting a KeyError message despite the public and private keys being correctly entered.

Here is my stack trace.

File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__return self.wsgi_app(environ, start_response)
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/matt/work/bsureunion/run.py", line 71, in contact
if form.validate_on_submit():
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask_wtf/form.py", line 166, in validate_on_submit
return self.is_submitted() and self.validate()
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/wtforms/form.py", line 310, in validate
return super(Form, self).validate(extra)
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/wtforms/form.py", line 152, in validate
if not field.validate(self, extra):
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/wtforms/fields/core.py", line 200, in validate
Display the sourcecode for this frameOpen an interactive python shell in this framestop_validation = self._run_validation_chain(form, chain)
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/wtforms/fields/core.py", line 220, in _run_validation_chain
validator(form, self)
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask_wtf/recaptcha/validators.py", line 47, in __call__
if not self._validate_recaptcha(response, remote_ip):
File "/home/matt/work/bsureunion/venv/lib/python2.7/site-packages/flask_wtf/recaptcha/validators.py", line 74, in _validate_recaptcha
for error in json_resp["error-codes"]:
KeyError: 'error-codes'

forms.py

from flask.ext.wtf import Form, RecaptchaField
from wtforms import StringField, TextAreaField, SubmitField
from wtforms.validators import InputRequired, Email
from wtforms.fields.html5 import EmailField

class ContactForm(Form):
name = StringField("Name", validators=[InputRequired('Please enter your name.')])
email = EmailField("Email",  validators=[InputRequired("Please enter your email address."), Email("Please enter your email address.")])
subject = StringField("Subject", validators=[InputRequired("Please enter the subject.")])
message = TextAreaField("Message", validators=[InputRequired("Please enter your message.")])
recaptcha = RecaptchaField()
submit = SubmitField("Send")

run.py

import os
from flask import Flask, render_template, request, flash
from forms import ContactForm
from flask.ext.mail import Message, Mail

mail = Mail()

app = Flask(__name__)

app.secret_key = 'secret_'

app.config["MAIL_SERVER"] = "smtp.gmail.com"
app.config["MAIL_PORT"] = 465
app.config["MAIL_USE_SSL"] = True
app.config["MAIL_USERNAME"] = 'me@gmail.com'
app.config["MAIL_PASSWORD"] = 'password'

SECURITY_EMAIL_SENDER = 'me@gmail.com'

RECAPTCHA_USE_SSL = False
RECAPTCHA_PUBLIC_KEY = 'public'
RECAPTCHA_PRIVATE_KEY = 'private'
RECAPTCHA_OPTIONS = {'theme': 'white'}

mail.init_app(app)

****snip****

@app.route('/contact/', methods=['GET', 'POST'])
def contact():
form = ContactForm()

if request.method == 'POST':
    if form.validate() == False:
      flash('All fields are required.')
      return render_template('contact.html', form=form)
if form.validate_on_submit():
    msg = Message(form.subject.data, sender='contact@bsureunion.com', recipients=['me@gmail.com'])
    msg.body = """
    From: %s <%s>
    %s
    """ % (form.name.data, form.email.data, form.message.data)
    mail.send(msg)

    return render_template('contact.html', success=True)

elif request.method == 'GET':
  return render_template('contact.html', form=form)

contact.html

{% extends "layout.html" %}

{% block body %}
<h2>Contact</h2>

{% if success %}
  <p>Thank you for your message. We'll get back to you shortly.</p>
  <p>Return to <a href="{{ url_for('home') }}">Home Page</a>.</p>

{% else %}

  {% for message in form.name.errors %}
    <div class="flash">{{ message }}</div>
  {% endfor %}

  {% for message in form.email.errors %}
    <div class="flash">{{ message }}</div>
  {% endfor %}

  {% for message in form.subject.errors %}
    <div class="flash">{{ message }}</div>
  {% endfor %}

  {% for message in form.message.errors %}
    <div class="flash">{{ message }}</div>
  {% endfor %}

  <form action="{{ url_for('contact') }}" method=post>
    {{ form.hidden_tag() }}

    {{ form.name.label }}
    {{ form.name }}

    {{ form.email.label }}
    {{ form.email }}

    {{ form.subject.label }}
    {{ form.subject }}

    {{ form.message.label }}
    {{ form.message }}

    {{ form.recaptcha }}

    {{ form.submit }}
  </form>

{% endif %}
{% endblock %}

OSError: [Errno 13] Permission denied:

I am trying to install a library package to python but I get the following error:

OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/PackageName'

I've looked online and tried the following, but it doesn't seem to work:

chown -R $USER /Library/Python/2.7/site-packages/pync

and also:

chown -R $vdixon /Library/Python/2.7

django listing files instead of showing home page

I'm not able to figure out what's going wrong. I'm using pycharm and bitnami django stack for developing my first web application.

Here is my directory structure:

project name: myapp

location: C:\Bitnami\djangostack-1.7.8-0\apache2\htdocs\myapp

Directory structure:

myapp
    manage.py
    app
          admin.py
          models.py
          settings.py
          tests.py
          urls.py
          views.py
          wsgi.py

          migrations
    templates
          home.html

my settings.py has following data:

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'app',
)

ROOT_URLCONF = 'app.urls'

TEMPLATE_DIRS = (
    os.path.join(BASE_DIR, 'templates'),
)

my urls.py has following data:

from django.conf.urls import url

from . import views

urlpatterns = [
    url(r'myapp/', views.homepage, name='home'),
]

my views.py has following data:

from django.http import HttpResponse


def homepage(request):
    return HttpResponse("Hello, world!")

Now when I try to run:

http://localhost/myapp

It simply displays the list of files in the myapp directory

I'm not able to find why it is not executing from urls.py

get URL of any submission for subreddits

I am trying to use PRAW to get new posts from subreddits on Reddit. The following code snippet shows how I get new items on a particular subreddit.

Is there a way to also get the URL of the particular submission?

submissions = r.get_subreddit('todayilearned')
submission = submissions.get_new(limit=1)
sub = [str(x) for x in submission]
print sub

Scrapy throws ImportError: cannot import name xmlrpc_client

After install Scrapy via pip, and having Python 2.7.10:

scrapy
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 7, in <module>
from scrapy.cmdline import execute
File "/Library/Python/2.7/site-packages/scrapy/__init__.py", line 48,  
in <module>
from scrapy.spiders import Spider
File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py",    
line 10, in <module>
from scrapy.http import Request
File "/Library/Python/2.7/site-packages/scrapy/http/__init__.py", line   
12, in <module>
from scrapy.http.request.rpc import XmlRpcRequest
File "/Library/Python/2.7/site-packages/scrapy/http/request/rpc.py",  
line 7, in <module>
from six.moves import xmlrpc_client as xmlrpclib
ImportError: cannot import name xmlrpc_client

But I can import module:

Python 2.7.10 (default, Jun 10 2015, 19:42:47) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
>>> 

What's going on?

Error: Nothing returns when running which python on terminal

I'm new to Python and I'm trying to install it. When I do $ brew install python Warning: python-2.7.10 already installed But when I put which python It returns nothing Please help :) Thank you!

Threaded Python 3 proxy checker is massively spiking the load average on AWS EC2 instance

This is very peculiar...

I have written a threaded proxy checker in Python 3 using pycurl.

It works great, and chugs along sipping cpu cycles averaging at most 30% of CPU and 2% memory when viewed via the top command.

Until it doesn't...

The script keeps running and doing it's job, and the server is still responsive, but occasionally the load average as seen via the top command will quickly spike to 75+ for seemingly no reason, and then quickly drop down to 0.20-0.30.

wait and steal % never push above 1%, and the python3 script never goes above 30% CPU, either.

What could be causing the CPU load average to spike so high and then quickly drop back down?

It's certainly the python3 script causing the load average spikes because it never happens when the script isn't running, and I have watched the top command on this server for days if not weeks in the past few months.

Does EC2 simulate the load average differently that other servers, perhaps?

Usually when the load average spikes above say 5.0 for more than a few minutes... as it has in the past when I was writing to MySQL too frequently... everything stops. Wait and steal% go through the roof, and I can barely even log into the server via SSH.

This load average seems fake to me, could it be?

I have copied the results of a top command below to show you how strange this looks:

top - 23:43:58 up 4 days,  6:51,  2 users,  load average: 11.79, 4.12, 2.61
Tasks: 116 total,   2 running, 114 sleeping,   0 stopped,   0 zombie
%Cpu(s): 34.2 us,  0.7 sy,  0.0 ni, 64.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.3 st
KiB Mem:   2048532 total,  1484256 used,   564276 free,   139420 buffers
KiB Swap:        0 total,        0 used,        0 free.   674300 cached Mem

0.0 wait, virtually 0 steal%, 64% idle CPU and a load average through the roof... how can this be?

Is it bad form to count on exceptions?

My current situation is involved in making a game in python using a library called pygame. I have a method called getTile() that returns the tile every time the player moves. The lvlDict makes up the world.

def getTile(self, pos):
    x = pos[0]
    y = pos[1]
    return self.lvlDict[x,y].kind

and i'm thinking about changing it to this

def getTile(self, pos):
    try:
        x = pos[0]
        y = pos[1]
        return self.lvlDict[x,y].kind
    except KeyError:
        return None

where there would be some interpretation for what to do if it were None. Likely just move back to where it previously was. Is this inherently bad or acceptable? Even if it's just a matter of opinion, I'd like to know what people think about it.

How can I perform an in-place mathematical operation on a column in Python?

I am trying to perform feature scaling on the values in an array. Given array X, I want to scalar subtract the mean of each column from the value of each element, and then divide each elements by the column's standard deviation. Is there any way to do this without iterating over each value? This is my attempt:

import numpy as np

X_norm = X;
mu = np.zeros(len(X[:,0]))
sigma = np.zeros(len(X[:,0]))

for i in range(len(mu)):
    mu[i] = np.mean(X[i,:])
    sigma[i] = np.std(X[i,:])
    X_norm[i,:] = (X_norm[i,:] - mu[i])/sigma(i)

I receive

TypeError: 'numpy.ndarray' object is not callable

when I try to run it.

Handling image from Google Maps APIs (staticmap)

I am trying to get an image from GoogleMaps APIs, more precisely from the staticmap API.
The problem is that in other APIs from GoogleMaps you can choose wether you want your info from the API in JSON or XML, but with staticmap (which returns an image) it seems you can't. So I don't know how to handle the image provided by the URL since I don't know how it is coded.

This is what I´m trying to do:

import requests

url = ("http://ift.tt/1Km9kHo")
response = requests.get(url)
print(response.json())

Hope you've got any advice about how to turn the response into something usable

Add string to a Numpy Array

I have a given list 'l' and given string 's', How can I combine them as shown in the required answer?

import numpy as np

l = [(1, 2),
     (3, 4),
     (5, 6)]

s = 'ax'

The required answer is:

ans = array([[ax, 1, 2],
             [ax, 3, 4],
             [ax, 5, 6]])

I tried as:

ans = np.column_stack([s, l])

wrong!

Transpose Census Flat Files in Python

I'm trying to transpose this US Census flat file: http://ift.tt/1Km9j6v in Python.

In the first column, the first 14 characters represent a row, and the last three represent a column. The second column is the value of that column and row. Can't seem to figure out a good way to make this into a table using Python.

Side Note: My end goal is to create a script that automatically imports these sort of files into ArcGIS, that's why I'm trying to do this in Python.

Grab data from Yahoo Finance using Meteor Package, some work some do not

I am using the following package for Meteor http://ift.tt/1Ich9ON

I cant seem to get a specified field to work, here is a link that contains a list of all the available fields, however 'j2' and some others I tested don't work, in the sense there is no response in the result object, or no json key pair values.

Heres is my client side code.

Template.stock.rendered = function (){
    if ( _.isEmpty(Session.get('ENW.V')) ) {
        Meteor.call('getQuote', 'ENW.V', function(err, result) {
            Session.set('ENW.V', result['ENW.V']);
            console.log(result)
        });
    }
}


Template.stock.helpers({
    stock: function() {
        return Session.get('ENW.V');
    }
})

Server side Method

Meteor.methods({
  getQuote: function( stockname ) {
    return YahooFinance.snapshot({symbols: [stockname] , fields:['n','a','b','j2'] });
  }
});

Thanks for any Help in Advance. Happy to add any additional info if needed.

having problems parsing gson

I am trying to parse data from here http://ift.tt/1GJqrvW

in particular this part

RATES: {
 RS: [
      {
          BEG: "12:00 AM",
          END: "2:00 AM",
          RATE: "0",
          RQ: "No charge"
      },
      {
          BEG: "2:00 AM",
          END: "6:00 AM",
          RATE: "0",
          RQ: "Str sweep"
      },
      {
          BEG: "6:00 AM",
          END: "12:00 AM",
          RATE: "0",
          RQ: "No charge"
      }
  ]
},

and I have this code

if (dataObject.getAsJsonObject("RATES").isJsonObject()){
  JsonObject rates = dataObject.getAsJsonObject("RATES");
  if (rates.getAsJsonArray("RS").isJsonArray()){
      //parse stuff
  }
}

I get com.google.gson.JsonObject cannot be cast to com.google.gson.JsonArray in here

 if (rates.getAsJsonArray("RS").isJsonArray())

and If I change that to

if (rates.getAsJsonObject("RS").isJsonObject())

I get com.google.gson.JsonArray cannot be cast to com.google.gson.JsonObject

Deserialize JSON with Numeric Rows

I am trying to deserialize some JSON into a list using JSON.NET; however, there is a number in the way:

Here is the JSON:

"payment_info": {
    "fb_id": "",
    "order_items": {
      "0": {
        "product_id": "4534",
        "type": "product",
        "shipping_cost_per_item": "1.00",
        "quantity": "3",
        "price_each": "10.00",
        "price_total": "30.00"
      }
    },

Here is my class:

    public class OrderItem
    {
        public string product_id { get; set; }
        public string type { get; set; }
        public string shipping_cost_per_item { get; set; }
        public string quantity { get; set; }
        public string price_each { get; set; }
        public string price_total { get; set; }
    }

    public class OrderItems
    {
        public List<OrderItem> Items { get; set; }
    }

How do I tell the converter to ignore the 0? There will be a 1,2,3 for each order item.

How to cancel a scheduled email in mandrill?

I am using http://ift.tt/1DRHqds package to handle my emails in mandrill. The package has a method to schedule an email like this.

$timestamp = new DateTime('+1 hour');
$mandrill = MailTo::Mandrill();
$mandrill->addRecipient($email, $name)
         ->setFrom($email, $name)
         ->setHtml($html)
         ->setText($text)
         ->setSubject($subject)
         ->send($timestamp);

But I can't find a way to cancel a scheduled email. I read this docs http://ift.tt/1dMiZt6

Request JSON

 {
    "key": "example key",
    "id": null
} 

but I don't know how to implement this. Does anyone can help me with this?

How to read latitude and longitude from json and plot it on google maps

I want to read latitude and longitude from json, the json file looks like this, and get plot the latitude and longitude on the google map in android. The latitude is name and longitude is country.

[{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"},{"name":"13.0714562","country":"77.55946348","twitter":"Current Location"}]

Here is my android maps code.

public class Maps extends Activity {
      private static String url = "http://ift.tt/1kpxaCr";
       static final LatLng College = new LatLng(13.1172245 , 77.6341758);

       private GoogleMap googleMap;
       float lat,lon;

       @Override
       protected void onCreate(Bundle savedInstanceState) {
          super.onCreate(savedInstanceState);
          setContentView(R.layout.maps);
          try { 


                if (googleMap == null) {
                   googleMap = ((MapFragment) getFragmentManager().findFragmentById(R.id.map)).getMap();
                    googleMap.moveCamera(CameraUpdateFactory.newLatLngZoom(College, 15));

                }

             googleMap.setMapType(GoogleMap.MAP_TYPE_NORMAL);

             @SuppressWarnings("unused")
            Marker TP = googleMap.addMarker(new MarkerOptions().position(College).title("Revamp 15,click on the arrow below for directions"));


          } catch (Exception e) {
             e.printStackTrace();
          }

       }

    }

How should i replace the static final LatLng College = new LatLng(13.1172245 , 77.6341758); witht the json data ?

How to return an array instead of object in JSON response?

I've one working REST API developed using Slim PHP framework.

It's working absolutely fine.

The only issue is when there is no error present i.e. an array $errors is empty it comes as an array in JSON response but when the array $errors contains any error like $errors['user_name'] then it comes as an object in JSON response.

Actually I want to return the array when error is present. How should I do this? Can someone please help me in this regard?

Thanks in advance.

access anonymous property from object

I have a set of nesting json object like this:

var obj = {
  name: "student",
  contact: {
     phone: "22222",
     fax: "33333",
     ...
  },
  ...
}

And I have these text fields:

<input type="text" name="name" />
<input type="text" name="contact_phone" />
<input type="text" name="contact_fax" />
...

Now I want to fill these fields with appropriate property from above object. My question is how can I access anonymous property from that object? For example suppose I have this jquery code:

 $("#formID").find("input").each(function(index) {
    fieldName = $(this).attr("name");
    var namePart = fieldName.split("_");
    //I want something like this: console.log(obj.namePart[0].namePart[1])
});