New IBM mainframes

A lot of noise from IBM about new Z mainframe product with new CPs, IO and several features, coupled with DS8000 storage. Good news for old mainframe addicts.

This time they have called it z13 (previous is z196 and z10 before).

Still being excellent platform but it makes less and less sense using it compared to open platforms.

BTW, 13 is not so lucky number in many countries…

Big Data solutions overview

‘Big Data’ is popular topic today. Actually it is quite generic term used for extra large data volumes analysis involving some special technics and specific solutions to handle such big sizes. Solutions can be devided into 2 classes:

  1. Solutions for structured big data (DBMS)
  2. Solutions for unstructured big data (web pages, images, logs etc), aka MapReduce (e.g. Hadoop)

Major part of big data solutions use MPP architecture with ‘shared nothing’ paradigm (local data), excluding Oracle Exadata which use shared database (special Oracle RAC solution).

Also 2 types of solutions could be realized:

  1. Software (DB/Hadoop platforms plus clustering facilities), which allow building solution on generic hardware;
  2. Integrated solution which binds sw/hw stack and allows only select combinations.

Hybrid solution are also available from some vendors (actually those are just 2 solutions)

List of solutions (Forrester and Gartner researches with some additions, some can be outdated for now) are given below.


Solution Type, summary Vendor


Cloudera CDH,

Cloudera Enterprise

Most popular distribution of Apache Hadoop platform and commercial management tools Cloudera


Teradata Aster MapReduce Platform, Teradata Database (only as part of Appliance) Distributed database for massively parallel environment, MapReduce/BigData TeraData/Aster Data


Teradata Appliance (Data Mart, Datawarehouse, Extreme),

Teradata Aster MapReduce Appliance

Integrated solution (appliance), Appliance – RDBMS,  Aster – RDBMS/Hadoop


Oracle Database Relational DBMS with DWH/OLAP extensions (columnar storage etc) Oracle


Oracle Exadata Integrated solution (appliance) using Oracle Database in RAC configuration


Oracle BigData Appliance Integrated solution (appliance) using Hadoop (Cloudera CDH)


Greenplum Database Distributed database for massively parallel environment (software) EMC/Greenplum


Greenplum HD Integrated solution (appliance) using Hadoop


Greenplum DCA Hybrid Integrated solution (appliance) -

GreenPlum Database + HD (Hadoop)


Sybase IQ Relational DBMS optimzed  for DWH/OLAP (s/w) SAP / Sybase


ParAccel Database Distributed database for massively parallel environment (software), can be integrated with Hadoop ParAccel


InfoSphere WareHouse Relational DBMS (DB2) optimized for DWH/OLAP IBM


Netezza Integrated DBMS (distributed) solution  (appliance), uses FPGA on nodes for extra speed


Infobright Community Edition (ICE) – open source , Infobright Enterprise Edition (IEE) DBMS optimized  for DWH/OLAP (s/w) Infobright


SQL Server 2008 R2 Parallel Data Warehouse Distributed DBMS for DWH/OLAP Microsoft


Microsoft FastTrack Integrated solution (appliance) using SQL Server PDW


Vertica Distributed DBMS for DWH/OLAP Vertica/HP


VectorWise Distributed DBMS for DWH/OLAP Actian (Ingres)


WX2 Distributed DBMS for DWH/OLAP, in-memory Kognitio


Kognitio Analytical Appliance Solution using WX2

Example architectures

IBM Netezza (MPP DBMS)

Teradata Aster (MapReduce/Hadoop)


Hybrid Solution (Paraccel)



Choose yours by price, functionality, support and complexity criterias…

Common analytics platform architecture

Business Intelligence/Analytics systems are more and more demanded by companies. Generic Information Analytical system architecture looks like this:


ETL level – rules and tools to get different data types for analysis.

DWH level – storage of data in form prepared for analytical queries, common views /marts

Data analysis level –BI, Search and OLAP systems are here

Types of platforms/components for each level  are to be covered in later posts.

Enterprise I/O interfaces summary

1. PCI-Express (PCIe).

Most commonly interface for IO connection to computer system (via IO controller to CPU and memory). Speed and bandwidth defined by number of lanes and speed of each lane. Maximum number of lanes also define adapter and slot form-factor (x4, x8, x16 etc). Downward compatible  at least in theory :) . Current version is 3.0.

PCIe version

Number of lanes








PCIe 1.0








PCIe 2.0








PCIe 3.0








2. Other most commonly used IO interfaces (external). All of them most of the time use PCIe at physical level to connect to computer system (CPU/mem), while at least Infiniband could be used for direct connection (was developed as an alternative to PCI). While there are many others (HDMI, DVI,  serial, RS232 etc) they are used today in very specific areas and are beiing displaced by commodity standards like USB.

Type Primary purpose Speed/bandwidth of modern generation

1 Gigabit Ethernet

10 Gigabit Ethernet

40 Gigabit Ethernet

100 Gigabit Ethernet

Network communication between computer systems, access to shared resources, including storage

(100GbE for end to end communication not available, planned for ~2016)

Adapted versions for hi-speed storage access exist (FcoE/DCE, also considered as ‘converged’)


1 Gbit/s

10 Gbit/s

40 Gbit/s

100 Gbit/s

Fiber Channel (FC) Storage networking, storage access, inter storage communication. 8 Gbit/s

16 Gbit/s

32 Gbit/s

FICON Mainframe communication, connection of mainframes to storage 4 Gbit/s

8 Gbit/s

16 Gbit/s

(FC incapsulation)


SAS 1.0 (SAS 3G)

SAS 2.0 (SAS 6G)

SAS 3.0 (SAS 12G)

Storage interface, most commonly used in today enterprise storage (direct system access, disk access etc.). Modern SCSI.

Compatible with SATA


3 Gbit/s

6 Gbit/s

12 Gbit/s


SATA 2.0

SATA 3.0

Storage access interface, most commonly used in low-cost and SOHO/consumer storage products 3 Gbit/s

6 Gbit/s






Hi-speed and low-latency network protocol. Mostly used for intersystems communications (in HPC clusters etc) and massively parallel storage access in cluster environments.  

20 Gbit/s (IB 4X DDR)

40 Gbit/s (IB 4X QDR)

56 Gbit/s (IB 4X FDR)

100 Gbit/s / c (IB 4X EDR)


USB 2.0

USB 3.0

General purpose interface for peripheral devices connections to computer system.  

400 Mbit/s

5 Gbit/s

Couple of notes about Fujitsu Eternus storage

We used Fujitsu Eternus for citrix vdi project in DR config. While it’s not so common as HP, IBM, Netapp and HDS here is a couple notes to memories.

  1. Eternus is standard block level midrange SAN array. No fancy modern vraids, virtualization etc, but standard features like thin provisioning, standard RAID levels, replication (Remote Equivalent Copy, quite good), snapshots are there. Actually for majority of tasks relative simplicity is not bad, given the price is adequate.
  2. A little bit strange logic in web gui interface, but still can be quite quickly understood.
  3. Special software is needed to configure remote replication. Unfortunately, replication (advanced copy) configuration is not available from web interface. In that case you need to install additional software to some host – management suite (SF) with advanced copy manager, which requires some rare third-party database product to function (symfoware or something like this). That’s quite painful, if you are not do this on daily basis. Actually you need this sw only for lun pairing, then you can use web gui to start-stop-reverse etc. So it looks quite silly to install complex sw to perform 1-2 tasks, hope this will be integrated to web gui. Still this sw needed if you are going to use AdvancedCopy to perform backups, then you will have AC agents, which will communicate with AC manager.
  4. While it’s japanese product, considered to be good quality, not sure about super-reliability. We have had 2-3 disks and 1 controller faults in 2-3 month project timeframe. Again it could be specific case. I think today almost all products are compareable in quality. So you need service. For storage you need good service. If Fujitsu service (specifically eternus) in your region is not so good, don’t buy Eternus. Or buy at least a couple of systems ).
  5. Docs are quite detailed, but not so clear and usefull, topics are distributed to many documents, some topics not covered etc.

I would say it’s just OK given you have service support.

See other info on Fujitsu resources.

100Gb/s is here

Infiniband EDR  switches from Mellanox are already available, making inter-server communications much more faster. That’s really cool. The need in hi-end servers continues to die. Commodity hardware connected by hi-speed network will be everywhere.

Old SUN Microsystems (RIP) paradigm ‘network is the computer’ comes closer and closer to reality.