We are almost exclusively a user of HP Procurve switches where I work. We have a wide range of models that we’ve used at both core and edge and have been happy with them over the years.

One important part of our current toolset for managing switches is the HP Procurve Manager Plus management tool (another part is the invaluable tips on the evil routers website). Once you move beyond a certain amount of switches it becomes inefficient to manage them all by hand and tools like this which allow you to bulk manage error and performance logging and manage switches in bulk become invaluable.

pcm screen

I installed the latest version of the PCM+ Server and client around December 2012 and things appeared to function ok. We installed it onto a multihomed server, that is, a server with more than one active network card. This is a supported configuration for Procurve Manager, though there is one important thing noted in the release notes:
When ProCurve Manager (either client or server) starts up, it attaches itself to the primary network interface. All network traffic between the client and server will be directed to the selected network interface. For example, if the ProCurve Manager client application attaches itself to the 166.3.4.5 interface, and the ProCurve Manager server is running on the 15.255.120.* network, there is no way that the client will ever connect successfully to the server.

To resolve this problem PCM has a configuration file that you can change to correct this situation. To setup this file, follow these steps:

Find the commIpAddr.txt file. This file exists in the config directory, so for example, for the client this file exists in: C:\Program Files\Hewlett-Packard\PCM\client\config . For the PCM server, you need to create a text file (with Notepad or similar application) and name it “commIpAddr.txt” and place it in the C:\Program Files\Hewlett-Packard\PCM\server\config directory.

Edit the file with a text-based editor (such as Notepad or Wordpad), and enter the IP address of the interface you want the application to attach to. For example for the network illustrated above, you would add the entry “ 15.255.120.25 " (without the quotes) in the first line of the file. More than one IP address can be used, but each IP Address entry must be on a separate line.

Save your changes.

Restart the application. If this is the ProCurve Manager client, just restart the application. If this is the ProCurve Manager server, you must restart the PCM services (HP ProCurve -Datastore, -Network Manager Server, and -Traffic Launch Service) from the Services control panel.

Got all that? Good. This is an important example of why reading the install notes prior to running setup is a good idea, at least for server apps, as there isn’t a lot of stuff out there on the web telling you about this, unless you already know what to look for.

If you don’t make this change then the Procurve Server will bind itself to the first IP address on the first network card it can see. Procurve Manager client that the installer just placed on your server will work as it is connecting ‘locally’ to your PCM server and can use any and all IP addresses it wants to do so, but clients which cannot reach that IP address will fail to get a connection and simply claim they cannot see the server.

So, ever mindful of this, I made the change described to our server. I created an entry in c:\program files(x86)\Hewlett Packard\PCM\server\config\commipaddr.txt and our PCM manager clients connected to the server just fine.

The switches themselves connected fine too, because they use SNMP to communicate with the PCM server and this is easily configured by entering something like snmp-server host 10.10.10.10 community “myswitches” into the switch’s configuration.

The problem came when trying to run the PCM+ software update module. This is a routine that allows you to download switch firmware updates to a central repository on the PCM+ server from the HP website and then install the firmware updates onto your switches automatically via a scheduled task.

This utterly failed for us. We had an error returned saying that the switches (which were on a series of 10.x.x.x/24 subnets) could not connect to the PCM server on 192.168.171.14. This subnet is part of our seperate iSCSI network and initially I was quite surprised to see this. While I knew the PCM+ server had a connection to this network (it had other monitoring software installed, including something that monitored the health of the iSCSI switches and the SANs). I double-checked the settings on the PCM server. I double-checked my commipaddr.txt file was correctly located and configured, and I checked all other bindings I could think of on the server. Nothing.

Clearly the software update routine was still somehow binding to the iSCSI network adapter, but this was a supported scenario and I had configured PCM+ correctly to work around this.

I opened a case with HP support, and spoke to a very knowledgeable engineer named Derek who ran through my switch, PCM+ client and PCM+ server configs with me and agreed they were correct. Derek went away to talk to a few of his colleagues. From what they were saying, the majority of calls they get about PCM+ boil down to “halp, where’s the ‘any’ key” and “No, I didn’t read the pre-install notes. What’s a commipaddr.txt file / other known issue” and this call generated a lot of interest from the HP support team who appeared to relish dealing with such a weird problem at last.

Anyway, after some hunting and painfully repetetive troubleshooting steps on both sides we finally found and fixed the issue, and I hope this helps anyone else having a similar problem in the future.

HP PCM+ is a modular system. Even when you opt to just install the PCM+ server, this is broken down into components:

  • Procuve Manager Plus client - what you use to connect to the Procuve Manager Plus server. This is what you use to view switch configurations and make changes to them. Obvious right?
  • Procurve Manager Plus Server - this manages collection of data from and distribution of instructions to your switches. This manages the PCM+ database and provides an central management point for clients and switches to connect to. Again, still obvious.
  • Procuve Manager Agent - this is the third part of the triumvirate. The agent is the service that actually collects the data and actually pushes out your configuration changes to switches.
The problem was with the agent.

As mentioned we had changed the Procuve Manager Plus server’s bindings through commipaddr.txt during installation. Changing the client’s bindings isn’t needed because it takes its cue from the server component its connecting to. It can either see the server or it cannot, and if it can then it will connect with no problem.

But the agent is a seperate component from both of these, and this isn’t really documented very well. If you wish to change the agent as well as the server, you need to create (or copy from the server component’s folders) another commipaddr.txt file. This needs to be placed in the root (not the /config folder) of the PCM Agent install point (e.g. C:\Program Files (x86)\Hewlett-Packard\PNM\pcm-agent\commipaddr.txt) and the PCM server service needs to be restarted.

With that done, repeat your troubleshooting step to ensure that connectivity is still available and test the software update again. This should now work, and if it does not, reboot the PCM+ server.