<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Bruce,<br>
<br>
On 23/09/13 17:55, Bruce B wrote:<br>
</div>
<blockquote
cite="mid:CAJyE_uUTmYsd=W15viEuZdcj-R1H8iRDM4D5f=s=UZ+SzJ9VaA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Eneko,</div>
<div><br>
</div>
VMs are off and results are - I think it was off before too (I
have some CentOS containers that are on which I can't turn off -
production!):
<div><br>
</div>
<div>
<div><font color="#0000ff">CPU BOGOMIPS: 72530.72</font></div>
<div><font color="#0000ff">REGEX/SECOND: 589160</font></div>
<div><font color="#0000ff">HD SIZE: 94.49 GB
(/dev/mapper/pve-root)</font></div>
<div><font color="#0000ff">BUFFERED READS: 100.20 MB/sec</font></div>
<div><font color="#0000ff">AVERAGE SEEK TIME: 11.14 ms</font></div>
<div><font color="#0000ff">FSYNCS/SECOND: 19.79</font></div>
</div>
</div>
</blockquote>
<font color="#0000ff">Definitely this value is too low. I've never
seen such a low value, even in nodes with working VMs and slower
hard disks! :)</font><br>
<br>
Can you mount a sdb partition a run pveperf on that disk? If both
disks are the same model, but values differ, then we could conclude
that it's a sda disk problem.<br>
<blockquote
cite="mid:CAJyE_uUTmYsd=W15viEuZdcj-R1H8iRDM4D5f=s=UZ+SzJ9VaA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div><font color="#0000ff">DNS EXT: 74.88 ms</font></div>
</div>
<div><font color="#0000ff"><br>
</font></div>
<div><font color="#000000">I am feeling the pain on Windows big
time but nothing bad on Containers. </font><b><font
color="#ff0000">So far we don't have a conclusion if it's
the kernel issue, HDD issue, or controller issue right?</font></b></div>
<div><b><font color="#ff0000"><br>
</font></b></div>
<div><b><font color="#ff0000"><br>
</font></b></div>
<div><font color="#000000">Info asked is below:</font></div>
<div><font color="#000000"><br>
</font></div>
<div>
<div><font color="#0000ff">root@hp:~# lspci</font></div>
<div><font color="#0000ff">00:00.0 Host bridge: Intel
Corporation 5520 I/O Hub to ESI Port (rev 13)</font></div>
<div><font color="#0000ff">00:01.0 PCI bridge: Intel
Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1
(rev 13)</font></div>
<div><font color="#0000ff">00:03.0 PCI bridge: Intel
Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3
(rev 13)</font></div>
<div><font color="#0000ff">00:07.0 PCI bridge: Intel
Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7
(rev 13)</font></div>
<div><font color="#0000ff">00:09.0 PCI bridge: Intel
Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root
Port 9 (rev 13)</font></div>
<div><font color="#0000ff">00:0a.0 PCI bridge: Intel
Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root
Port 10 (rev 13)</font></div>
<div><font color="#0000ff">00:14.0 PIC: Intel Corporation
7500/5520/5500/X58 I/O Hub System Management Registers
(rev 13)</font></div>
<div><font color="#0000ff">00:14.1 PIC: Intel Corporation
7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers
(rev 13)</font></div>
<div><font color="#0000ff">00:14.2 PIC: Intel Corporation
7500/5520/5500/X58 I/O Hub Control Status and RAS
Registers (rev 13)</font></div>
<div><font color="#0000ff">00:1a.0 USB controller: Intel
Corporation 82801JI (ICH10 Family) USB UHCI Controller #4</font></div>
<div><font color="#0000ff">00:1a.7 USB controller: Intel
Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2</font></div>
<div><font color="#0000ff">00:1c.0 PCI bridge: Intel
Corporation 82801JI (ICH10 Family) PCI Express Root Port 1</font></div>
<div><font color="#0000ff">00:1c.4 PCI bridge: Intel
Corporation 82801JI (ICH10 Family) PCI Express Root Port 5</font></div>
<div><font color="#0000ff">00:1d.0 USB controller: Intel
Corporation 82801JI (ICH10 Family) USB UHCI Controller #1</font></div>
<div><font color="#0000ff">00:1d.1 USB controller: Intel
Corporation 82801JI (ICH10 Family) USB UHCI Controller #2</font></div>
<div><font color="#0000ff">00:1d.2 USB controller: Intel
Corporation 82801JI (ICH10 Family) USB UHCI Controller #3</font></div>
<div><font color="#0000ff">00:1d.7 USB controller: Intel
Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1</font></div>
<div><font color="#0000ff">00:1e.0 PCI bridge: Intel
Corporation 82801 PCI Bridge (rev 90)</font></div>
<div><font color="#0000ff">00:1f.0 ISA bridge: Intel
Corporation 82801JIR (ICH10R) LPC Interface Controller</font></div>
<div><font color="#0000ff">00:1f.2 SATA controller: Intel
Corporation 82801JI (ICH10 Family) SATA AHCI Controller</font></div>
</div>
</div>
</blockquote>
<br>
This seems to be the controller. Never used this integrated
controller with proxmox, but I don't think it's the reason for low
fsyncs...<br>
<br>
Cheers<br>
<br>
<blockquote
cite="mid:CAJyE_uUTmYsd=W15viEuZdcj-R1H8iRDM4D5f=s=UZ+SzJ9VaA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div><font color="#0000ff">02:00.0 VGA compatible controller:
Matrox Electronics Systems Ltd. MGA G200e [Pilot]
ServerEngines (SEP1) (rev 02)</font></div>
<div><font color="#0000ff">05:00.0 Ethernet controller: Intel
Corporation 82576 Gigabit Network Connection (rev 01)</font></div>
<div><font color="#0000ff">05:00.1 Ethernet controller: Intel
Corporation 82576 Gigabit Network Connection (rev 01)</font></div>
<div style="color:rgb(0,0,0)"><br>
</div>
<div style="color:rgb(0,0,0)">Regards,</div>
</div>
<div>
<font color="#000000"><br>
</font></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Sep 23, 2013 at 11:09 AM, Eneko
Lacunza <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:elacunza@binovo.es" target="_blank">elacunza@binovo.es</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Bruce,<br>
<br>
pveperf is quite bad. From my limited experience, a
tipical 7200 rpms SATA drive gives 60 fsync/s and
>100 MB/sec buffered reads. Average seek time is very
bad too (~13ms in 7200 rpm drive). If you had VMs
running, please stop them all and rerun the command.<br>
<br>
You shouldn't used this for virtualization unless this
problem is fixed (you're already feeling the pain eh??)<br>
<br>
What hard disk controller do you have? ('lspci')
<div>
<div class="h5"><br>
<br>
On 23/09/13 16:30, Bruce B wrote:<br>
</div>
</div>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">Eneko,
<div><br>
</div>
<div>Thanks for the feedback. It seems that the
whole Windows system is slow. It happens with
loading applications too and loading start menu
for example so if I am understanding this right,
viritio drives which are installed after Windows
is installed may not help me a lot?! Please
correct me if I am wrong. Also how can I build a
virtio drive to test it?</div>
<div><br>
</div>
<div><b>Below are results of pveperf. Is this very
bad?</b></div>
<div><br>
</div>
<div>
<div><font color="#0000ff">root@hp:~# pveperf</font></div>
<div><font color="#0000ff">CPU BOGOMIPS:
72530.72</font></div>
<div><font color="#0000ff">REGEX/SECOND:
583443</font></div>
<div><font color="#0000ff">HD SIZE:
94.49 GB (/dev/mapper/pve-root)</font></div>
<div><font color="#0000ff">BUFFERED READS:
61.13 MB/sec</font></div>
<div><font color="#0000ff">AVERAGE SEEK TIME:
29.30 ms</font></div>
<div><font color="#0000ff">FSYNCS/SECOND:
9.63</font></div>
<div><font color="#0000ff">DNS EXT:
70.07 ms</font></div>
<div><br>
</div>
</div>
<div> Regards,</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Sep 23, 2013 at
3:00 AM, Eneko Lacunza <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:elacunza@binovo.es"
target="_blank">elacunza@binovo.es</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Bruce,<br>
<br>
pveperf on the disk (/) ?<br>
<br>
If you haven't, I think it will help you a
lot installing virtio drivers on the
Windows guest, then changing VM disks from
ide to virtio.
<div>
<div><br>
<br>
On 22/09/13 22:20, Bruce B wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">Thanks for feedback
Krzysztof and Alexandre. Below are
the info:
<div><br>
</div>
<div>I am using 1x 500GB WD HDD. I
can add another one if that helps
- something like: WD5001AALS.
Would that help? Where do you read
the IOPS? and what is a good
number of IOPS today?</div>
<div><br>
</div>
<div>For VM I am using LOCAL QCOW2 -
not sure how virtio drives work.</div>
<div><br>
</div>
<div>Hoping following info help you
tell me if I am hitting a
controller bottleneck (meaning I
can't help it) or if it is an HDD
problem:</div>
<div><br>
</div>
<div>
<div><font color="#0000ff">
*-storage</font></div>
<div><font color="#0000ff">
description: SATA
controller</font></div>
<div><font color="#0000ff">
product: 82801JI (ICH10
Family) SATA AHCI Controller</font></div>
<div><font color="#0000ff">
vendor: Intel Corporation</font></div>
<div><font color="#0000ff">
physical id: 1f.2</font></div>
<div><font color="#0000ff">
bus info: <a
moz-do-not-send="true"
href="mailto:pci@0000:00:1f.2"
target="_blank">pci@0000:00:1f.2</a></font></div>
<div><font color="#0000ff">
logical name: scsi0</font></div>
<div><font color="#0000ff">
version: 00</font></div>
<div><font color="#0000ff">
width: 32 bits</font></div>
<div><font color="#0000ff">
clock: 66MHz</font></div>
<div><font color="#0000ff">
capabilities: storage msi
pm ahci_1.0 bus_master
cap_list emulated</font></div>
<div><font color="#0000ff">
configuration:
driver=ahci latency=0</font></div>
<div><font color="#0000ff">
resources: irq:50
ioport:d880(size=8)
ioport:d800(size=4)
ioport:d480(size=8)
ioport:d400(size=4)
ioport:d080(size=32)
memory:faffc000-faffc7ff</font></div>
<div><font color="#0000ff">
*-disk</font></div>
<div><font color="#0000ff">
description: ATA Disk</font></div>
<div><font color="#0000ff">
product: WDC
WD5001AALS-0</font></div>
<div> <font color="#0000ff">
vendor: Western
Digital</font></div>
<div><font color="#0000ff">
physical id: 0.0.0</font></div>
<div><font color="#0000ff">
bus info: <a
moz-do-not-send="true"
href="mailto:scsi@0:0.0.0"
target="_blank">scsi@0:0.0.0</a></font></div>
<div><font color="#0000ff">
logical name: /dev/sda</font></div>
<div><font color="#0000ff">
version: 05.0</font></div>
<div><font color="#0000ff">
serial:
WD-WCATR2413417</font></div>
<div><font color="#0000ff">
size: 465GiB (500GB)</font></div>
<div><font color="#0000ff">
capabilities:
partitioned partitioned:dos</font></div>
<div><font color="#0000ff">
configuration:
ansiversion=5 sectorsize=512
signature=00064f12</font></div>
<div> <font color="#0000ff">
*-volume:0</font></div>
<div><font color="#0000ff">
description: EXT3
volume</font></div>
<div><font color="#0000ff">
vendor: Linux</font></div>
<div><font color="#0000ff">
physical id: 1</font></div>
<div><font color="#0000ff">
bus info: <a
moz-do-not-send="true"
href="mailto:scsi@0:0.0.0,1"
target="_blank">scsi@0:0.0.0,1</a></font></div>
<div><font color="#0000ff">
logical name:
/dev/sda1</font></div>
<div><font color="#0000ff">
logical name: /boot</font></div>
<div><font color="#0000ff">
version: 1.0</font></div>
<div><font color="#0000ff">
serial:
8fe2447e-4258-4d39-b7c7-450b66460abf</font></div>
<div><font color="#0000ff">
size: 511MiB</font></div>
<div><font color="#0000ff">
capacity: 511MiB</font></div>
<div><font color="#0000ff">
capabilities:
primary bootable journaled
extended_attributes recover
ext3 ext2 initialized</font></div>
<div><font color="#0000ff">
configuration:
created=2013-08-06 16:38:36
filesystem=ext3
modified=2013-08-09 17:14:18
mount.fstype=ext3
mount.options=rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered
mounted=2013-08-09 17:14:18
state=mounted</font></div>
<div><font color="#0000ff">
*-volume:1</font></div>
<div><font color="#0000ff">
description: Linux
LVM Physical Volume partition</font></div>
<div><font color="#0000ff">
physical id: 2</font></div>
<div><font color="#0000ff">
bus info: <a
moz-do-not-send="true"
href="mailto:scsi@0:0.0.0,2"
target="_blank">scsi@0:0.0.0,2</a></font></div>
<div><font color="#0000ff">
logical name:
/dev/sda2</font></div>
<div><font color="#0000ff">
serial:
pqnaJf-WL5Q-kz2z-a3CJ-f6rS-Jxz4-LICwof</font></div>
<div><font color="#0000ff">
size: 465GiB</font></div>
<div><font color="#0000ff">
capacity: 465GiB</font></div>
<div><font color="#0000ff">
capabilities:
primary multi lvm2</font></div>
</div>
<div><br>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra">Thanks,<br>
<br>
<div class="gmail_quote">On Sun,
Sep 22, 2013 at 6:07 AM,
Krzysztof Bloniarz <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:kb0spam@gmail.com"
target="_blank">kb0spam@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hi Bruce,
<div><br>
</div>
<div>Could you confirm that
you are using one 500GB
SATA drive as your storage
?</div>
<div>How many VMs are
running on this drive ?</div>
<div><br>
</div>
<div>This SATA drive is
capable of 60 IOPS maybe
70IOPS, you can easily
saturate this installing
windows apps, particularly
if you run simultaneous
VMs on that drive </div>
<div><br>
</div>
<div>To solve your
'performace' problems you
have to build RAID and add
more spindles.</div>
<div><br>
</div>
<div>Regards,</div>
<div>KB</div>
<div> </div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
<div>
<div> On Fri, Sep 20,
2013 at 8:33 PM, Bruce
B <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:bruceb444@gmail.com" target="_blank">bruceb444@gmail.com</a>></span>
wrote:<br>
</div>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
<div>
<div dir="ltr">Hi
Everyone,
<div><br>
</div>
<div>I am seeing
very slow
install of
applications
within a Windows
2008R2 VM that I
built with 24Gbs
of RAM (no users
on it yet) and
the Proxmox
server is a
DL160 G6 with
Dual L5520 Xeon
quad core CPUs.
I don't see why
this is acting
so slow. I am
looking for
suggestions on
how to make this
work faster. </div>
<div><br>
</div>
<div>Below are my
findings of IO
stats and HDD
specifications.
I would like to
know if there is
any hope to this
server. I am
running Windows
2008 R2 in IDE0
and QCOW mode.</div>
<div><br>
</div>
<div>
<div><font
color="#ff0000"><b>root@hp:~#
iostat -xkd 2
(util
shows over 97%
below as a
program is
being
installed)</b></font></div>
<div>Linux
2.6.32-22-pve
(hp)
09/20/2013
_x86_64_
(16 CPU)</div>
<div><br>
</div>
<div>Device:
rrqm/s
wrqm/s r/s
w/s
rkB/s
wkB/s
avgrq-sz
avgqu-sz
await r_await
w_await svctm
%util<br>
</div>
<div>sda
0.00
13.50 0.00
94.50
0.00 9023.25
190.97
2.41 25.43
0.00 25.43
10.32 97.50</div>
<div>dm-0
0.00
0.00 0.00
13.00
0.00 52.00
8.00
0.70 54.15
0.00 54.15
6.77 8.80</div>
<div>dm-1
0.00
0.00 0.00
0.00
0.00 0.00
0.00
0.00 0.00
0.00 0.00
0.00 0.00</div>
<div>dm-2
0.00
0.00 0.00
95.00
0.00 8913.25
187.65
2.29 24.12
0.00 24.12
10.24 97.30</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><font
color="#ff0000"><b>hdparm
output:</b></font></div>
<div>
<pre style="margin-top:4px;margin-bottom:20px;width:800px;overflow:auto;font-size:9pt;font-family:verdana;line-height:18pt;padding:15px;border:2px solid rgb(255,174,0);background-color:rgb(252,242,221);color:rgb(136,136,136)">
ATA device, with non-removable media
Model Number: WDC WD5001AALS-00E3A0
Serial Number: WD-WCATR2413417
Firmware Revision: 05.01D05
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6
Standards:
Supported: 8 7 6 5
Likely used: 8
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 976773168
Logical/Physical Sector size: 512 bytes
device size with M = 1024*1024: 476940 MBytes
device size with M = 1000*1000: 500107 MBytes (500 GB)
cache/buffer size = unknown
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 0
Recommended acoustic management value: 128, current value: 254
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
Power-Up In Standby feature set
* SET_FEATURES required to spinup after power up
SET_MAX security extension
Automatic Acoustic Management feature set
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* 64-bit World wide name
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Native Command Queueing (NCQ)
* Host-initiated interface power management
* Phy event counters
* NCQ priority information
* DMA Setup Auto-Activate optimization
* Software settings preservation
* SMART Command Transport (SCT) feature set
* SCT Long Sector Access (AC1)
* SCT LBA Segment Access (AC2)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
unknown 206[12] (vendor specific)
unknown 206[13] (vendor specific)
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
supported: enhanced erase
102min for SECURITY ERASE UNIT. 102min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee2af8fec40
NAA : 5
IEEE OUI : 0014ee
Unique ID : 2af8fec40</pre>
<pre style="margin-top:4px;margin-bottom:20px;width:800px;overflow:auto;font-size:9pt;font-family:verdana;line-height:18pt;padding:15px;border:2px solid rgb(255,174,0);background-color:rgb(252,242,221);color:rgb(136,136,136)">
</pre>
<pre style="margin-top:4px;margin-bottom:20px;width:800px;overflow:auto;font-size:9pt;font-family:verdana;line-height:18pt;padding:15px;border:2px solid rgb(255,174,0);background-color:rgb(252,242,221);color:rgb(136,136,136)">
Thanks,</pre>
</div>
</div>
<br>
</div>
</div>
<div>_______________________________________________<br>
pve-user mailing list<br>
<a
moz-do-not-send="true"
href="mailto:pve-user@pve.proxmox.com" target="_blank">pve-user@pve.proxmox.com</a><br>
<a
moz-do-not-send="true"
href="http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user"
target="_blank">http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user</a><br>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
pve-user mailing list
<a moz-do-not-send="true" href="mailto:pve-user@pve.proxmox.com" target="_blank">pve-user@pve.proxmox.com</a>
<a moz-do-not-send="true" href="http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user" target="_blank">http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user</a>
</pre>
</blockquote>
<br>
<br>
</div>
</div>
<span><font color="#888888">
<pre cols="72">--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943575997
943493611
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
<a moz-do-not-send="true" href="http://www.binovo.es" target="_blank">www.binovo.es</a></pre>
</font></span></div>
<br>
_______________________________________________<br>
pve-user mailing list<br>
<a moz-do-not-send="true"
href="mailto:pve-user@pve.proxmox.com"
target="_blank">pve-user@pve.proxmox.com</a><br>
<a moz-do-not-send="true"
href="http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user"
target="_blank">http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
<pre cols="72">--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943575997
943493611
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
<a moz-do-not-send="true" href="http://www.binovo.es" target="_blank">www.binovo.es</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943575997
943493611
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
<a class="moz-txt-link-abbreviated" href="http://www.binovo.es">www.binovo.es</a></pre>
</body>
</html>