Getting customized mobile updates from twitter and other RSS services

This post marks the 4th year of this blog.
Hope is to actively engage in writing this blog.

Presented below is an orchestrate of seemingly disparate technologies to enable your favorite updates on your cellphone.
Note: Your cellphone need not have Wifi or GPRS options. This works on basic SMS incoming facility.

Now some of my favorite online items which i need to keep track are:
1) Twitter updates from selected friends.
2) Posts from selected blogs which i want to keep myself updated.
3) Instant updates from Google alerts.

Here is one way we can have all the elements integrated:

1) Collect RSS feeds of all the items listed above.
2) Using yahoo pipes you can do the following:
2a) Using Fetch Feed , aggregate all the feeds.
2b) Use a sort by published date to get the feeds in order. Click here for a preview
2c) This service gives out a common aggregated RSS feed for the selected set.
2d) Also one can use alternatives like RSSmix to achieve a common RSS feed.
3) To get the updates on the cellphone, use Google SMS Channel . By configuring our cellphone for the alerts in Google SMS Channel , we can get the desired updates on our cellphone.

Java URL Bloomer

I came across an interesting problem in java



public class URLBloomer {
public static void main(String[] args) throws MalformedURLException, URISyntaxException {
String[] urls= {
"http://meshlabsinc.com",
"http://meshserver.org"
};
Set testing=new HashSet();
for (String url : urls) {
testing.add(new URL(url));
}
System.out.println(testing.size());
}
}


Output : 1

Reason .... meshlabsinc.com and meshserver.org point to the same IP


Lesson learnt is to use URI instead of URL class.


public class URLBloomer {
public static void main(String[] args) throws MalformedURLException, URISyntaxException {
String[] urls= {
"http://meshlabsinc.com",
"http://meshserver.org"
};
Set testing=new HashSet();
for (String url : urls) {
testing.add(new URI(url));
}
System.out.println(testing.size());
}
}

Output : 2

Getting the create table command for an already existing table in mysql

mysql>SHOW CREATE TABLE

mysql> create table x (i integer);
Query OK, 0 rows affected (0.27 sec)

mysql> show create table x;
+-------+--------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+--------------------------------------------------------------------------------------+
| x | CREATE TABLE `x` (
`i` int(11) default NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1 |
+-------+--------------------------------------------------------------------------------------+
1 row in set (0.06 sec)

mysql>

Java code to send message to your gtalk friends

Have used a library called Smack (http://www.igniterealtime.org/projects/smack/index.jsp)

import java.util.Collection;

import org.jivesoftware.smack.*;

public class GtalkClient {

public static void main(String[] args) throws XMPPException {
ConnectionConfiguration config = new ConnectionConfiguration("talk.google.com", 5222, "gmail.com");
XMPPConnection connection = new XMPPConnection(config);
connection.connect();
connection.login("< username >","< password >");

// Below is the code to get the users


// Roster roster = connection.getRoster();
// Collection rosterEntries = roster.getEntries();
//
// System.out.println("\n\n" + rosterEntries.size() + " friend(s):");
// for(RosterEntry rosterEntry:rosterEntries)
// {
// System.out.println(rosterEntry.getUser());
// }

// Here is a code to send message to a friend

MessageListener messageListener = null;
Chat chat = connection.getChatManager().createChat("shantanu.gg@gmail.com",messageListener);
chat.sendMessage("Hello this is a ping from a java program");
}
}

Sending email using GMAIL using Java Mail Api

Note: Search for changeme and replace with appropriate terms.





import javax.mail.*;
import javax.mail.internet.*;
import java.util.*;

public class MailSender {
final String senderEmailID = "changeme @ changeme .com";
final String senderPassword = "changeme";
final String emailSMTPserver = "smtp.gmail.com";
final String emailServerPort = "465";
String receiverEmailID = null;
String emailSubject = null;
String emailBody = null;

public MailSender(String receiverEmailID, String emailSubject, String emailBody) {
this.receiverEmailID=receiverEmailID;
this.emailSubject=emailSubject;
this.emailBody=emailBody;


Properties props = new Properties();
props.put("mail.smtp.user",senderEmailID);
props.put("mail.smtp.host", emailSMTPserver);
props.put("mail.smtp.port", emailServerPort);
props.put("mail.smtp.starttls.enable", "true");
props.put("mail.smtp.auth", "true");
// props.put("mail.smtp.debug", "true");
props.put("mail.smtp.socketFactory.port", emailServerPort);
props.put("mail.smtp.socketFactory.class",
"javax.net.ssl.SSLSocketFactory");
props.put("mail.smtp.socketFactory.fallback", "false");

SecurityManager security = System.getSecurityManager();

try {
Authenticator auth = new SMTPAuthenticator();
Session session = Session.getInstance(props, auth);
// session.setDebug(true);

MimeMessage msg = new MimeMessage(session);
msg.setText(emailBody);
msg.setSubject(emailSubject);
msg.setFrom(new InternetAddress(senderEmailID));
msg.addRecipient(Message.RecipientType.TO,
new InternetAddress(receiverEmailID));
Transport.send(msg);
} catch (Exception mex) {
mex.printStackTrace();
}


}
private class SMTPAuthenticator extends javax.mail.Authenticator {
public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(senderEmailID, senderPassword);
}
}
/**
* @param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
MailSender mailSender=new MailSender("changeme @ changeme .com","Test Mail from Puretechie","Here goes the body");
}

}

Power Vs (LVM Vs VxVm). Score 2-0. Power Wins :(

we had an unplanned/unscheduled power outage recently and as a result of that, some servers went down, some storage controllers went down, some switches were up and some local disks went bad.

chaos, you may think.
it was actually much more challenging than that. and every such situation in office gives a good learning experience. once the storage controllers, switches and power to serves were restored, we had this problem.

there is one each imaging server for HP-UX, Solaris and AIX environments respectively(NOT imaging as in picture/photos imaging, this is a hard disk imaging environment). we can also call it as network based installation technologies for each h/w architecture type. All these servers store the disk images in a centralised archive server(which was, in our case, the same as the imaging server for hp-ux). the storage came from a MSA with one LUN of 2 TB presented to this host.

the server wudnt boot up after the power outage from the local disk (which was mirrored). the kernel wud just dump core and reboot infinitely. I and my colleagues were not great hp-ux kernel debuggers, so we took the alternate steps:

1. Boot from mirrored disk (although we know that a mirrored disk doesnt give data redundancy cos if data on one disk is corrupted, it wud almost simultaneously corrupt data on the other disk) - FAILED. Problem cud be hardware (memory, cpu ... although leds were all green) or OS related.
2. since the server (rp*) had 3 slots for scsi disks, we attached a disk from another server on which hp-ux (version immaterial at this point) was installed and booted the failed server from this disk. System Booted: h/w is OK. what could be so wrong with the OS that it dumps so bad??? and a power outage caused this?!?
3. we dint know the OS environment on the disk that had gone bad(version, file system type, partition information).

core options we had were
hp-ux 11.23 or 11.31
with
LVM or VxVm.

thats a combo of four.

and if u screw up with the metadata(fs data), u r totally lost. not just the 2TB data, configs and other data from the local disk(which had no backup ;) ) wud simply be wiped out. so, the strategy was to recover the local disk data(somehow) and then to recover the data on MSA.

a quick look at docs said LVM and VxVm totally co-exist and are "aware" of each other. good news, also bad. good news, cos they are aware of each other and hence wud _probably_ not tamper with the meta-data wen u try to access them. bad news, cos they co-exist, and hence our number of combinations to be tried out is still 4.

with fingers crossed, and since the third disk which had currently booted the server had 11.31 with lvm, i thought i will try the lvm method.

run sam, go to disks and filesystems menu, locate ur disk. note down the hardware path.
go back to volume groups menu, and say "import volume group".
u get a msg quickly saying that LVM data is not found on disk. phew.
this could mean disk corruption OR that it is a vxvm disk.

try the command line tools that sam just used with the force options vgscan, vgimport and u get same error.
OK. so its a vxvm disk.(trying to be optimistic)

apparently, vxvm software is installed as part of every hp-ux installation(may be only internally for us, cos vxvm is a licensed tool) but not activated unless you use it for root dg(disk group). vxvm and lvm have different terminologies.

LVM / VxVm
Physical Volume / VxVM Disk
Logical Volume / Volume
Volume Group / Disk Group
Physical Extent / Subdisk
LVM metadata / Private Region
Unused Physical / Free Space
Extent

LVM, as u see, is what we are mostly exposed to(due to linux and traditinal unix fs). so u know the terminologies and the equivalents. now, u need to know how to use the software itself (vxvm).
its quite easy. just run vxinstall. and use all default options. vxvm has a web based client for a front-end. the tool is called "vea" (Veritas Enterprise Administrator). vea just showed the physical disks. nothing else. all other entities(DGs, volumes) were empty. right-click on ur disk and say "recover disk" (with crossed fingers, ofcourse). and voila, the disk groups were imported. it was, a vxvm disk. root, swap and stand volumes were shows as detected, but corrupted. okay. so you run a fsck on the disk, correct all vxfs errors. and mount the root volume. ur config data and other data is there.

now for the data on the MSA. right click on msa disk and said "recover disk", just like earlier. and bam! i got a disk group that showed as 2TB free space!!! where is all the data!

here is where some concepts learnt in college helps. u most certainly know that u have only screwed up the meta-data and not yet the actual fs data. cos a disk was only initialised to be of some type(vxvm disk) and not formatted.

u go back to ur root disk and search for /mount-point/etc/fstab (damn, why dint i look into it before!!!) and it said, very promptly, that the disk was of LVM type (device names of vxvm disks are /dev/vx/dg-name/volumes, where as lvm will have it as /dev/vg-name/lvol-names). who wud think that the msa will be configured for lvm type and the root disk will be of vxvm type??? confirm it by running "strings /mount-point/etc/lvmtab" , the disk was sitting pretty there with the vg data.

ok, now, deactivate the disk using vea and delete all vxvm data. go back to sam and say "import volume group". as expected, lvm data was lost, so sam says, do u want to create a new volume group instead??? (hell, no!)

since i had access to the root disk, navigate to /mount-point/etc/lvmconfig/ and search for .conf file that will have the vg data.
using that, vgcfgrestore and vgimport commands, we cud restore the vg config. but the superblock was also corrupted(fsck said so). so had to dump the superblock from 8192 and successive(forgot the block count) blocks. fsck again and mount the lvol. phew, the data was all there!

now for the anti-climax:
even though we cud recover all data, we cudnt boot off the local disk that was dumping core. vxvm used to hang during boot-up right after initialising the disks. we had to re-setup the imaging server(we also tried using the "last_install" kernel and that wudnt boot either). a similar thing happened on an x86 VM (running on ESX 3.5)recently, which was also a critical server for us. even though fsck gave no errors and we cud access all data when mounted as an external disk on another VM, we could not boot from that disk. and thats when i give up and create the setup once again from scratch :( using the data recovered externally. and to top it all, what has power outage got to do with this???

PS: most of this might not be clear due to the language and the sequence in which i have put it. leave ur quetions as comments.

Common bad practice in database calculations

create table x ( i int, j varchar(10))
insert into x values (1,'a')
insert into `x`(`i`,`j`) values ( '2',NULL)


select count(*) from x
2
Select count(j) from x
1

Note the difference when the count(col_name) is used.

count(col_name) is used under the impression that it is faster than count(*) , however quite the opposite is true.

MYISAM table MySQL has cached number of rows in the table. Thats the reason why MYISM is able to instantly answer COUNT(*) query, but not COUNT(col_name).

Why ? Because say if col_name column is not defined as NOT NULL there can be some NULL values in it and so MySQL have to perform table scan to find out. This is also why result is different for the second query.

Using count(*) instead of count(col_name) falls in the best practice category.