Обсуждение: LEFT JOIN and missing values
Hi, I have a problem, like this: I have 2 tables, T1 (id, val1) -- a vector and T2 (id1, id2, val2) -- this is actually a sqare matrix in the form of a very long table. I need to do some operations with them, such as take a dot-product of 2 rows of the matrix (encoded as T2), weighted with T1.. . The query is the following: select c.id1, d.id1, sum(c.val2*d.val2*T1.val1) from T1, T2 c, T2 d where c.ID2 = T1.id AND d.id2 = T1.id AND c.id1< d.id1 group by c.id1, d.id1; This gives me all possible pairs of (weighted) dot-products, as I want them. Now, some 90% of values in the T2 are the same, say, 0.5, so I could thouw them away. But then the query above is not valid anymore, since if there is mo match on the condition c.ID2 = T1.id AND d.id2 = T1.id, nothing is added (it might be that c.id2 is there and d.id2 is not!). I quiess I need a left join + substitute for missing value. Can anyone help me with the query? I cannot figure it out, too bad ad left joins... thanks, Mila
mila wrote: > Hi, > > I have a problem, like this: > > I have 2 tables, T1 (id, val1) -- a vector > and T2 (id1, id2, val2) -- this is actually a sqare matrix > in the form of a very long table. > > I need to do some operations with them, such as take a > dot-product of 2 rows of the matrix (encoded as T2), weighted with T1.. . > > The query is the following: > > select c.id1, d.id1, sum(c.val2*d.val2*T1.val1) > from T1, T2 c, T2 d > where c.ID2 = T1.id AND d.id2 = T1.id AND c.id1 < d.id1 > group by c.id1, d.id1; > > This gives me all possible pairs of (weighted) dot-products, as I want them. > > Now, some 90% of values in the T2 are the same, say, 0.5, so I could > thouw them away. But then the query above is not valid anymore, since > if there is mo match on the condition c.ID2 = T1.id AND d.id2 = > T1.id, nothing is added (it might be that c.id2 is there and d.id2 is > not!). > > I quiess I need a left join + substitute for missing value. Can anyone > help me with the query? I cannot figure it out, too bad ad left > joins... > > thanks, > Mila First let's rewrite your query to explicit joins (I like them very much, because they show the problem more clearly) select c.id1, d.id1, sum(c.val2*d.val2*T1.val1) from T1 join T2 c on (c.ID2 = T1.id) join T2 d on (d.id2 = T1.id and c.id1 < d.id1) group by c.id1, d.id1; If I understand well you want to delete useless values: delete from t2 where val2=0.5 your query should look then: select c.id1, d.id1, sum( case when c.id2 is null then 0.5 else c.val2 end * case when d.id2 is null then 0.5 else d.val2end *T1.val1) from T1 left join T2 c on (c.ID2 = T1.id) left join T2 d on (d.id2 = T1.id and (c.id1 < d.id1 or c.id1 is null)) group by c.id1, d.id1; I'm not sure about this query... There is a problem - is it possible to have missing values in your matrix? This solution replaces all missing values into 0.5. What should happen to "T2 alias d" if there is missing value in "T2 alias c"? What happens then to clause "c.id1<d.id1" if "c" is not found? Regards, Tomasz Myrta
Tomasz, > select c.id1, d.id1, sum( > case when c.id2 is null then 0.5 else c.val2 end * > case when d.id2 is null then 0.5 else d.val2 end *T1.val1) > from > T1 > left join T2 c on (c.ID2 = T1.id) > left join T2 d on (d.id2 = T1.id and (c.id1 < d.id1 or c.id1 is null)) > group by c.id1, d.id1; > I'm not sure about this query... There is a problem - is it possible to > have missing values in your matrix? This solution replaces all missing > values into 0.5. yes, that's what I want to do, remove the tuples with default value and substitute it later if I need it. > What should happen to "T2 alias d" if there is missing > value in "T2 alias c"? if a value, c.val2, is missing, it should be replaced with 0.5, since the only deleted tuples are those that had contained 0.5 in "val2" field. The (other) value, d.val2, that exisits in the matrix, should be used "as is". If they are both missing, then t1.val1 is multiplied twice by 0.5 > What happens then to clause "c.id1<d.id1" if "c" > is not found? I removed this condition "c.id1 < d.id1", since it is indeed unclear, what to compare when a row is "missing" (I can filter it later). Then, I tried the query that you proposed, on a test table T1 - 10 rows, T2 = 10x10 = 100 rows. With the full T2 the query is equivalent to the one that I gave, but when I remove the tuples with default values (call it table T3), then the resulting table is only 90 rows instead of 100, besides, the values are different from the "right" ones from "full" T2. The missing combinations seem to be random. How can that be if T1 contains all values that might be in T2.id2, T2.id1? and what do do next? Mila
I looked at your example once again and I noticed that wrong result comes from "group by" clause. If we delete useless rows with value "0.5", we also remove information about possible values of t2.id1. Let's say your table t2 is full of 0.5 values. It means, we can empty it. If we do this, how Postgres would know which t2.id1 values to return? What size of matrix should it return? I think if you really need to remove your 0.5 values, you need another table with all possible values of id1. This table will look like: create table t3 ( id1 integer; ); your query should look then: select cc.id1, dd.id1, sum( case when c.id2 is null then 0.5 else c.val2 end * case when d.id2 is null then 0.5 else d.val2end *T1.val1) from T1 cross join T3 cc left join T2 c on (c.ID2 = T1.id and c.id1=cc.id1) cross join T3 dd left join T2 d on (d.id2 = T1.idand d.id1=dd.id1) group by cc.id1, dd.id1; Anyway I'm not sure if it is what you really want :-( Regards, Tomasz Myrta
Tomasz, > I think if you really need to remove your 0.5 values, you need another > table with all possible values of id1. This table will look like: > create table t3 > ( > id1 integer; > ); T1 contains values of id's, The T2 matrix is a matrix of some sort of "distances" between id's from T1, so original T2 is a T1 x T1. (that's what you are proposing right?) So there is no id1, id2 in T2 that don't exists in T1, I checked my test tables and this is indeed the case. In other words T1 is dense, I make a left join on it and I get wrong results! And even wrong number of tuples :-( cheers, Mila