Parsing Python dictionary with multiple objects

Post by anuragpatibandla7
{"1":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"2":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"3":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"4":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"}}
Now if I have 100 objects like these and I need to split them into 3 smaller dicts in a ratio 2:3:5, how could I do that?

I really have no idea what that means. You have 100 dicts of
dicts? Are the keys unique? If so, you could combine them with a
loop of update.

Post by anuragpatibandla7
I tried using numpy.random.choice(), but it says it needs to be a 1-d array.

How about random.choice? It needs a sequence.

To get anything more concrete, you need to specify Python version,
and make a clearer problem statement, perhaps with example of
what you expect.

--
DaveA

Anurag Patibandla

2014-10-14 21:15:36 UTC

I really have no idea what that means. You have 100 dicts of
dicts? Are the keys unique? If so, you could combine them with a
loop of update.

Post by anuragpatibandla7
I tried using numpy.random.choice(), but it says it needs to be a 1-d array.

How about random.choice? It needs a sequence.
To get anything more concrete, you need to specify Python version,
and make a clearer problem statement, perhaps with example of
what you expect.
--
DaveA

Hey DaveA,
I am using Python 2.7.
Yes. I have a dict of dicts with keys ranging from 1..100
And the value of each of these keys is another dict. What I need to do is make 3 smaller dicts and assign the values of keys 1..100 in the ratio 2:3:5.
For example, if my original dict is

d={"1":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"2":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"3":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"4":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"}}
...
...
"100":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"}

I need to have three dicts d1, d2, d3 with d1 containing the values of first 20 keys, d2 containing the values on next 30 keys and d3 containing the values of the next 50.
Can you please help me with this?

Skip Montanaro

2014-10-14 21:32:38 UTC

Shuffle the keys, then grab the first 20 for one dictionary, the next 30
for the second, and the last 50 for the third.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20141014/ab00db13/attachment.html>

Anurag Patibandla

2014-10-14 21:36:49 UTC

Shuffle the keys, then grab the first 20 for one dictionary, the next 30 for the second, and the last 50 for the third.
Skip

Could you please be more specific?

MRAB

2014-10-14 22:58:25 UTC

I really have no idea what that means. You have 100 dicts of
dicts? Are the keys unique? If so, you could combine them with a
loop of update.

Post by anuragpatibandla7
I tried using numpy.random.choice(), but it says it needs to be a 1-d array.

How about random.choice? It needs a sequence.
To get anything more concrete, you need to specify Python version,
and make a clearer problem statement, perhaps with example of
what you expect.

Hey DaveA,
I am using Python 2.7.
Yes. I have a dict of dicts with keys ranging from 1..100
And the value of each of these keys is another dict. What I need to do is make 3 smaller dicts and assign the values of keys 1..100 in the ratio 2:3:5.
For example, if my original dict is
d={"1":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"2":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"3":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"},
"4":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"}}
...
...
"100":{"Key1":"Value1", "Key2":"Value2", "Key3":"Value3"}
I need to have three dicts d1, d2, d3 with d1 containing the values of first 20 keys, d2 containing the values on next 30 keys and d3 containing the values of the next 50.
Can you please help me with this?

You can get a list of the entries using the dict's .items method. It's
then a simple matter of slicing the list.

Note that dicts aren't ordered, i.e. its keys aren't in a fixed order,
so if you want then in a particular order, you'll need to sort the list.

Dave Angel

2014-10-15 01:57:08 UTC

What have you actually tried? You haven't shown any actual code.

Look up the method dict.keys, and see how you might use that. Then
look up random.shuffle, and see what it would do. Also look up
dict.sort, since your two messages on this thread imply two
conflicting goals as to which sub dictionaries should go in which
of your buckets.

As for extracting 20% of the keys, slicing is your answer. If
there are 100 keys, 20, 30, and 50 need to be sliced
off.

Then you'll need a loop to build each result dictionary from its keys.

There are shortcuts, but it's best to learn the fundamentals first.

Try writing the code. If it doesn?t all work show us what you've
tried, and what you think is wrong. And when you get an
exception, show the whole traceback, don't just paraphrase one
of the lines.

--
DaveA

Rustom Mody

2014-10-15 02:32:27 UTC

What have you actually tried? You haven't shown any actual code.
Look up the method dict.keys, and see how you might use that. Then
look up random.shuffle, and see what it would do. Also look up
dict.sort, since your two messages on this thread imply two
conflicting goals as to which sub dictionaries should go in which
of your buckets.
As for extracting 20% of the keys, slicing is your answer. If
there are 100 keys, 20, 30, and 50 need to be sliced
off.
Then you'll need a loop to build each result dictionary from its keys.
There are shortcuts, but it's best to learn the fundamentals first.
Try writing the code. If it doesn't all work show us what you've
tried, and what you think is wrong. And when you get an
exception, show the whole traceback, don't just paraphrase one
of the lines.

Yes that is what is in general expected out here -- code --
maybe working, maybe not, maybe incomplete, maybe 'pseudo' etc
Then others here will improve it

However there is one conceptual thing that perhaps should be mentioned: order.

Is your data *essentially* ordered?

And by 'essentially' I mean you think of it independent of python.
Yeah in python dicts are unordered and lists are ordered and one can fudge
one to behave a bit like the other. But before you fudge, please ponder which
you really need/want.

Below a bit of going from one to other

# dict -> list

d = {"a":1,"b":2,"c":3}
d

{'a': 1, 'c': 3, 'b': 2}

list(d)

['a', 'c', 'b']

# alternate

d.keys()

['a', 'c', 'b']

d.items()

[('a', 1), ('c', 3), ('b', 2)]

d.values()

[1, 3, 2]

# list -> dict

dict(d.items())

{'a': 1, 'c': 3, 'b': 2}

# round-tripping

dict(d.items()) == d

True

Anurag Patibandla

2014-10-15 03:40:40 UTC

Thanks for the response.
Here is the code that I have tried.

from operator import itemgetter
keys = json.keys()
order = list(keys)
q1 = int(round(len(keys)*0.2))
q2 = int(round(len(keys)*0.3))
q3 = int(round(len(keys)*0.5))
b = [q1,q2,q3]
n=0
for i in b:
queues = order[n:n+i]

n = n+i
print queues

for j in range(len(queues)):
q = (queues[j], json.get(queues[j]))
print q

By this I am able to get the 3 smaller dicts I want, but can you help me assign them to 3 variables?
The dicts need not be ordered but it would be better if they are ordered.

Thanks

Rustom Mody

2014-10-15 04:50:28 UTC

This post might be inappropriate. Click to display it.

Dave Angel

2014-10-15 17:43:58 UTC

Post by Anurag Patibandla
Thanks for the response.
Here is the code that I have tried.
from operator import itemgetter
keys = json.keys()
order = list(keys)
q1 = int(round(len(keys)*0.2))
q2 = int(round(len(keys)*0.3))
q3 = int(round(len(keys)*0.5))
b = [q1,q2,q3]
n=0

threedicts = []

Post by Anurag Patibandla
queues = order[n:n+i]
n = n+i
print queues
q = (queues[j], json.get(queues[j]))
print q

onedict = {}
for q in queues:
onedict[q] = json[q]
threedicts.append (onedict)
dict1, dictw, dict3 = threedicts

Post by Anurag Patibandla
By this I am able to get the 3 smaller dicts I want, but can you help me assign them to 3 variables?
The dicts need not be ordered but it would be better if they are ordered.

dicts are not ordered. If you want the items in a particular
order, you have to do that after extraction from the dict. There
is a related type called collections.OrderedDict, which
'remembers' the order things were added.

--
DaveA

Anurag Patibandla

2014-10-15 18:35:53 UTC

threedicts = []

Post by Anurag Patibandla
queues = order[n:n+i]
n = n+i
print queues
q = (queues[j], json.get(queues[j]))
print q

onedict = {}
onedict[q] = json[q]
threedicts.append (onedict)
dict1, dictw, dict3 = threedicts

Thanks DaveA!
This works perfectly!

Anurag Patibandla

2014-10-15 03:41:59 UTC

'json' has my original larger dict

Anurag Patibandla

2014-10-15 15:52:30 UTC

Thanks Rustom for the advice.
I am new to Python and getting struck at some basic things. How do I assign the values that I am printing to 3 variables say dict1, dict2, dict3?
When I try to assign them before the print statement like this:
d1, d2, d3 =[(queues[j], json.get(queues[j])) for j in range(len(queues))]

I get an error saying 'need more than one value to unpack'

Rustom Mody

2014-10-15 16:06:25 UTC

Post by Anurag Patibandla
Thanks Rustom for the advice.
I am new to Python and getting struck at some basic things. How do I assign the values that I am printing to 3 variables say dict1, dict2, dict3?
d1, d2, d3 =[(queues[j], json.get(queues[j])) for j in range(len(queues))]
I get an error saying 'need more than one value to unpack'

Probably means your comprehension

[(queues[j], json.get(queues[j])) for j in range(len(queues))]

is having less than 3 values

lst = [1,2,3]
x,y,z = lst
(x,y,z) # note no need to print

(1, 2, 3)

lst=[1]
x,y,z=lst

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: need more than 1 value to unpack

lst=[1,2,3,4]
x,y,z=lst

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: too many values to unpack
================

I suggest youdont directly start with multiple assignment
Instead do it in two steps

dicts = [(queues[j], json.get(queues[j])) for j in range(len(queues))]

d0 = dicts[0]
d1 = dicts[1]
d2 = dicts[2]

When that works go to the more compact form

Also please get rid of the range(len(queues))
Its unpythonic!

Anurag Patibandla

2014-10-15 16:28:38 UTC

First the values printed by
'[(queues[j], json.get(queues[j])) for j in range(len(queues))] '
is a list, so I tried to convert it into a dict using dict().
And then I tried doing dict[0] but there is an error which says:
'type' object has no attribute '__getitem__'

Rustom Mody

2014-10-15 16:43:06 UTC

Post by Anurag Patibandla
First the values printed by
'[(queues[j], json.get(queues[j])) for j in range(len(queues))] '
is a list, so I tried to convert it into a dict using dict().
'type' object has no attribute '__getitem__'

print each step in the process until the error step

Also assuming

[(queues[j], json.get(queues[j])) for j in range(len(queues))]

is the same as

[(q, json.get(q) for q in queues]

do

a = [(q, json.get(q) for q in queues]
print a
b = dict(a)
print b
c = b[0]
print c

or whatever it is you are doing

and PASTE (not NARRATE) the results

Rustom Mody

2014-10-15 16:59:12 UTC

Also there are dictionary comprehensions recently added to python:
http://legacy.python.org/dev/peps/pep-0274/

Can generally be used if you are doing
dict(a list comprehension)

To start with though, I suggest you stay with the longer older form
until you understand how it works

Anurag Patibandla

2014-10-15 16:57:28 UTC

keys = json.keys()
order = list(keys)
q1 = int(round(len(keys)*0.2))
q2 = int(round(len(keys)*0.3))
q3 = int(round(len(keys)*0.5))
b = [q1,q2,q3]
n=0
for i in b:
queues = order[n:n+i]

n = n+i
#print queues
#print [(queues[j], json.get(queues[j])) for j in range(len(queues))]
lists = [(queues[j], json.get(queues[j])) for j in range(len(queues))]
#print lists
dicts = dict(lists)
print dicts
print dict[0]

Print dicts works as expected giving me the combine dictionary values. But when I say dict[0]. I see the error:
TypeError: 'type' object has no attribute '__getitem__'

Dave Angel

2014-10-15 17:48:22 UTC

Post by Anurag Patibandla
dicts = dict(lists)
print dicts
print dict[0]
TypeError: 'type' object has no attribute '__getitem__'

Of course. You forgot the s in the name dicts. So you were
referring to the dict class, not your variable.

--
DaveA

Anurag Patibandla

2014-10-15 17:00:38 UTC

keys = json.keys()
order = list(keys)
q1 = int(round(len(keys)*0.2))
q2 = int(round(len(keys)*0.3))
q3 = int(round(len(keys)*0.5))
b = [q1,q2,q3]
n=0
for i in b:
queues = order[n:n+i]

n = n+i
lists = [(queues[j], json.get(queues[j])) for j in range(len(queues))]

dicts = dict(lists)
print dicts
print dict[0]

print dicts works as expected. It gives me the entire dictionary. But when I do dicts[0], there is the following error:
'type' object has no attribute '__getitem__'

Rustom Mody

2014-10-15 17:10:26 UTC

Post by Anurag Patibandla
keys = json.keys()
order = list(keys)
q1 = int(round(len(keys)*0.2))
q2 = int(round(len(keys)*0.3))
q3 = int(round(len(keys)*0.5))
b = [q1,q2,q3]
n=0
queues = order[n:n+i]
n = n+i
lists = [(queues[j], json.get(queues[j])) for j in range(len(queues))]
dicts = dict(lists)
print dicts
print dict[0]
'type' object has no attribute '__getitem__'

Do you want dict[0] ??
I think you want dicts[0]

Anurag Patibandla

2014-10-15 17:20:20 UTC

Do you want dict[0] ??
I think you want dicts[0]

Sorry about that.
dicts[0] gives me a KeyError: 0

Dave Angel

2014-10-15 17:50:54 UTC

Do you want dict[0] ??
I think you want dicts[0]

Sorry about that.
dicts[0] gives me a KeyError: 0

If the keys are all strings, why would you expect to find any
items with an int key?

--
DaveA

Anurag Patibandla

2014-10-15 17:20:57 UTC

Here is my sample dict if that helps:

json = {"1": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 1, "m_Quantile": "80", "m_Controller": "Python", "m_Method": "Distributed", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}, "3": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 3, "m_Quantile": "90", "m_Controller": "Python", "m_Method": "Distributed", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}, "2": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 2, "m_Quantile": "80", "m_Controller": "Python", "m_Method": "GARCH", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}, "4": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 4, "m_Quantile": "90", "m_Controller": "Python", "m_Method": "GARCH", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}}

Rustom Mody

2014-10-15 17:35:30 UTC

Post by Anurag Patibandla
json = {"1": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 1, "m_Quantile": "80", "m_Controller": "Python", "m_Method": "Distributed", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}, "3": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 3, "m_Quantile": "90", "m_Controller": "Python", "m_Method": "Distributed", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}, "2": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 2, "m_Quantile": "80", "m_Controller": "Python", "m_Method": "GARCH", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}, "4": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 4, "m_Quantile": "90", "m_Controller": "Python", "m_Method": "GARCH", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}}

Right
So your dict (which is dicts !) we have

Post by Anurag Patibandla
json.keys()

['1', '3', '2', '4']

And so

Post by Anurag Patibandla
json[0]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 0

Post by Anurag Patibandla
json['0']

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: '0'

Post by Anurag Patibandla
json['1']

{'Status': 'Submitted', 'Startdate': ['01/01/2011'], 'Enddate': ['02/02/2012'], 'Job_ID': 1, 'm_Quantile': '80', 'Allocation_3': ['50'], 'm_Method': 'Distributed', 'm_Controller': 'Python', 'Allocation_2': ['30'], 'Allocation_1': ['20'], 'Asset_2': ['YHOO'], 'Note': '', 'VaR': '', 'submit': ['Submit'], 'm_Iterations': '1000', 'Asset_3': ['CAT'], 'Asset_1': ['AAPL']}

IOW 0 is not a key
Neither is '0' (the string containing the char 0)
But the string '1' is a valid key

Anurag Patibandla

2014-10-15 18:27:18 UTC

Right
So your dict (which is dicts !) we have

Post by Anurag Patibandla
json.keys()

['1', '3', '2', '4']
And so

Post by Anurag Patibandla
json[0]

File "<stdin>", line 1, in <module>
KeyError: 0

Post by Anurag Patibandla
json['0']

File "<stdin>", line 1, in <module>
KeyError: '0'

Post by Anurag Patibandla
json['1']

{'Status': 'Submitted', 'Startdate': ['01/01/2011'], 'Enddate': ['02/02/2012'], 'Job_ID': 1, 'm_Quantile': '80', 'Allocation_3': ['50'], 'm_Method': 'Distributed', 'm_Controller': 'Python', 'Allocation_2': ['30'], 'Allocation_1': ['20'], 'Asset_2': ['YHOO'], 'Note': '', 'VaR': '', 'submit': ['Submit'], 'm_Iterations': '1000', 'Asset_3': ['CAT'], 'Asset_1': ['AAPL']}
IOW 0 is not a key
Neither is '0' (the string containing the char 0)
But the string '1' is a valid key

Yes, but I can't just do 'json['1']', at the end of the code I need to do a 'dicts['1']', or 'dicts['2']', to get the smaller dicts which still gives me a 'KeyError: 1'

Dave Angel

2014-10-15 22:25:19 UTC

e": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 2, "m_Quantile": "80", "m_Controller": "Python", "m_Method": "GARCH", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"],
"Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}, "4": {"Status": "Submitted", "Startdate": ["01/01/2011"], "Enddate": ["02/02/2012"], "Job_ID": 4, "m_Quantile": "90", "m_Controller": "Python", "m_Method": "GARCH", "Allocation_3": ["50"], "Allocation_2": ["30"], "Allocation_1": ["20"], "Note": "", "m_Iterations": "1000", "submit": ["Submit"], "VaR": "", "Asset_2": ["YHOO"], "Asset_3": ["CAT"], "Asset_1": ["AAPL"]}}

Post by Rustom Mody
Right
So your dict (which is dicts !) we have

Post by Anurag Patibandla
json.keys()

['1', '3', '2', '4']
And so

Post by Anurag Patibandla
json[0]

File "<stdin>", line 1, in <module>
KeyError: 0

Post by Anurag Patibandla
json['0']

File "<stdin>", line 1, in <module>
KeyError: '0'