openstack victoria 对接华为存储OceanStor 5110 v5（Kunpeng）
OpenStack VictoriaCinder 华为 Huawei OceanStor 5110 v5
·
1. 环境
本文主要侧重于victoria 版本对接华为存储OceanStor 5110 v5（Kunpeng）的步骤记录和问题记录，云平台的部署等相关问题本文不做介绍
HOST OS：Ubuntu 20.04.3 LTS
OpenStack：victoria （kolla），详情如下：
kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "victoria"
openstack_tag: "victoria"
存储：华为存储OceanStor 5110 v5（Kunpeng），万兆交换机
网络：具备管理、业务、存储等隔离网络条件
2. 华为存储配置
华为存储上需要做三项配置：
- 导入LICNESE文件，以支持cinder的对接
- 创建资源资源池，用于OpenStack cinder对接，其中主机组、LUN组、主机、映射等均不用做
- 配置管理地址和业务网络地址，分别用于API和业务使用
3. cinder配置
OpenStack的部署我们这里略过，直接介绍cinder配置相关信息，由于我们这块采用了kolla部署的openstack云平台，所以所有服务均运行于容器内，配置文件也通过映射模式映射到容器内的，配置文件（/etc/kolla/目录下）修改会有些差异，可根据各自的环境做适当调整。
github有华为openstack_cinder_driver配置文档：https://github.com/Huawei/OpenStack_Driver/tree/50ff55d2cad7d70bb4d26f5975326c2061556c41/ConfigDoc/zh
对接不同OpenStack版本的驱动代码：https://github.com/Huawei/OpenStack_Driver/tree/master/Cinder
cinder配置部分均在cinder-volume节点操作。
3.1 驱动下载导入
通过git clone https://github.com/Huawei/OpenStack_Driver.git 或者直接网页下载后解压，我们采用了当时最新的OpenStack_Driver-2.3.RC3
docker cp OpenStack_Driver-2.3.RC3 cinder_volume:/tmp
docker exec -it -u 0 cinder_volume bash
cd /tmp/OpenStack_Driver-2.3.RC3/Cinder/Victoria
mv * /var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/
3.2 cinder_huawei_conf.xml
创建如下文件，并编辑如下
vim /etc/kolla/cinder-volume/cinder_huawei_conf.xml
<?xml version='1.0' encoding='UTF-8'?>
<config>
<Storage>
<Product>V5</Product>
<Protocol>iSCSI</Protocol>
<RestURL>https://x.x.x.x:8088/deviceManager/rest/</RestURL>
<UserName>admin</UserName>
<UserPassword>******</UserPassword>
</Storage>
<LUN>
<LUNType>Thin</LUNType>
<LUNCopySpeed>4</LUNCopySpeed>
<StoragePool>xxx</StoragePool>
</LUN>
<iSCSI>
<Initiator HostName="*" ALUA="1" FAILOVERMODE="3" SPECIALMODETYPE="0" PATHTYPE="0" TargetIP="y.y.y.y z.z.z.z"/>
</iSCSI>
</config>
根据我们所采用的产品、协议和账号密码等填写Product、Protocol、RestURL，Username、Userpassword、StoragePool等信息，这些信息均在第二章所配置的。Userpassword项填写admin密码，RestURL项中的x.x.x.x填写管理地址，StoragePool项填写需要对接的资源池，TargetIP项填写业务地址。
3.3 config.json
添加cinder_huawei_conf.xml文件映射
vim /etc/kolla/cinder-volume/config.json
...
{
"source": "/var/lib/kolla/config_files/cinder_huawei_conf.xml",
"dest": "/etc/cinder/cinder_huawei_conf.xml",
"owner": "cinder",
"perm": "0700"
},
...
3.4 cinder.conf
采用iSCSI协议，enabled_backends以逗号分割，可以接多个后端，我们仅用一个，如下：
vim /etc/kolla/cinder-volume/cinder.conf
[DEFAULT]
enabled_backends = huawei_backend
[huawei_backend]
volume_driver = cinder.volume.drivers.huawei.huawei_driver.HuaweiISCSIDriver
cinder_huawei_conf_file = /etc/cinder/cinder_huawei_conf.xml
volume_backend_name = huawei_backend
use_multipath_for_image_xfer = true
enforce_multipath_for_image_xfer = true
做完以上配置，重启服务
docker restart cinder_volume
4. multipath配置
存储（cinder-volume）和计算（nova-compute）节点都需要配置
针对存储配置多路径
vim /etc/kolla/multipathd/multipath.conf
defaults {
user_friendly_names yes
find_multipaths yes
}
blacklist {
devnode "^sda$"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^sd[b-d]"
devnode "^nbd*"
}
devices {
device {
vendor "HUAWEI"
product "XSG1"
path_grouping_policy "group_by_prio"
path_checker tur
prio alua
detect_checker no
failback immediate
path_selector "round-robin 0"
dev_loss_tmo 30
fast_io_fail_tmo 5
}
}
5. 计算节点
配置nova-compute支持多路径及cpu_mode为透传
vim /etc/kolla/nova-compute/nova.conf
[libvirt]
volume_use_multipath = True
cpu_mode=host-passthrough
docker restart nova_compute
6. OpenStack使用后端存储
创建卷类型并关联后端如下：
cinder type-create huawei_backend
cinder type-key huawei_backend set volume_backend_name="huawei_backend"
通过cinder type-list获取存储后端信息，通过glance image-list获取镜像信息等：
cinder type-list
+--------------------------------------+----------------+---------------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+----------------+---------------------+-----------+
| 81a35e1a-70d5-4d19-9873-7c9a40cd7cda | huawei_backend | - | True |
+--------------------------------------+----------------+---------------------+-----------+
创建卷（启动卷含image id，亦可创建空卷）选择对应的后端：
cinder create --volume-type 81a35e1a-70d5-4d19-9873-7c9a40cd7cda--image-id 678f7dab-2c3b-44d3-9f9e-b758ddc1cbe8 --name volume-with-huawei 200
7. 问题记录
- 对接华为存储LICENSE invalid问题
参看cinder-volume.log日志详情如下：
2022-04-10 10:53:06.985 103 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg)
2022-04-10 10:53:06.985 103 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create lun {'NAME': '7636c1b0-3c7e9475b344d8138b0476', 'PARENTID': '3', 'DESCRIPTION': 'volume-7636c1b0-dcfa-4703-aaab-a806a3bd2ae7', 'ALLOCTYPE': '1', 'CAPACITY': 20971520, 'ENABLESMARTDEDUP': False, 'ENABLECOMPRESSION': False, 'MIRRORPOLICY': '1'} error.
2022-04-10 10:53:06.985 103 ERROR oslo_messaging.rpc.server result: {'data': {}, 'error': {'code': 1077951771, 'description': 'The license format is incorrect or the feature included in the license file is invalid.', 'suggestion': 'Download a valid license file. If the problem persists, contact technical support engineers.'}}.
解决办法：找华为存储厂商，要LICENSE，然后通过存储页面导入LICENSE文件
- 存储挂载Volume device not found at及重复initiator问题
nova-compute日志详情如下：
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line 68, in _wrapper
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] return r.call(f, *args, **kwargs)
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] do = self.iter(retry_state=retry_state)
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 368, in iter
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] raise retry_exc.reraise()
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 186, in reraise
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] raise self.last_attempt.result()
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] return self.__get_result()
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] raise self._exception
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 412, in call
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] result = fn(*args, **kwargs)
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 784, in _connect_multipath_volume
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] raise exception.VolumeDeviceNotFound(device='')
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]
2022-04-12 11:42:40.892 7 ERROR nova.volume.cinder [req-8036223c-164a-4bb4-9655-7889218a6567 5375c23f2bf445aa87b0518d3db820df 3e3013c5092b475980fd4e0fb9015bd7 - default default] Delete attachment failed for attachment dde284a2-1e4e-43c0-9610-6d9abea24fcb. Error: Volume attachment could not be found with filter: attachment_id = dde284a2-1e4e-43c0-9610-6d9abea24fcb. (HTTP 404) (Request-ID: req-0cbfdf63-5749-4302-a512-2aaf85c0002e) Code: 404: cinderclient.exceptions.NotFound: Volume attachment could not be found with filter: attachment_id = dde284a2-1e4e-43c0-9610-6d9abea24fcb. (HTTP 404) (Request-ID: req-0cbfdf63-5749-4302-a512-2aaf85c0002e)
2022-04-12 11:42:40.964 7 ERROR oslo_messaging.rpc.server [req-8036223c-164a-4bb4-9655-7889218a6567 5375c23f2bf445aa87b0518d3db820df 3e3013c5092b475980fd4e0fb9015bd7 - default default] Exception during message handling: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
cinder-volume.log日志详情如下：
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils conn = self.initialize_connection(volume, properties)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "<decorator-gen-763>", line 2, in initialize_connection
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/coordination.py", line 151, in _synchronized
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils return f(*a, **k)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/huawei_driver.py", line 70, in initialize_connection
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils self.configuration)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/huawei_flow.py", line 2475, in initialize_iscsi_connection
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils engine.run()
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils for _state in self.run_iter(timeout=timeout):
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils failure.Failure.reraise_if_any(er_failures)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils failures[0].reraise()
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 346, in reraise
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils six.reraise(*self._exc_info)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils raise value
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils result = task.execute(**arguments)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/huawei_flow.py", line 1444, in execute
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils initiator, host_id, alua_info)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/rest_client.py", line 613, in associate_iscsi_initiator_to_host
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils initiator, host_id)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/rest_client.py", line 93, in _assert_result
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils raise exception.VolumeBackendAPIException(data=msg)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Add initiator iqn.1994-05.com.redhat:92a5f47e3aa5 to host 25 error.
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils result: {'data': {}, 'error': {'code': 1077950338, 'description': 'The initiator has been added to another host.', 'suggestion': 'Specify a valid initiator and add it to the host.'}}.
解决办法：通过查看存储发现，挂载时候有两个启动器，且一个离线一个在线，绑定到离线的主机上了。而且由于是容器化部署，经过摸排发现是iscsid和nova_compute容器里面InitiatorName不一致导致的，每个计算节点又是同一个InitiatorName，故每个节点InitiatorName需要重新生成，且保证iscsid和nova_compute里面InitiatorName一致。
具体操作如下：
docker exec -it -u 0 iscsid bash
echo "InitiatorName=`/sbin/iscsi-iname`" > /etc/iscsi/initiatorname.iscsi
exit
docker cp iscsid:/etc/iscsi/initiatorname.iscsi /opt
docker cp /opt/initiatorname.iscsi nova_compute:/etc/iscsi/initiatorname.iscsi
docker restart iscsid nova_compute
- 基于镜像创建卷signature_verified问题
执行从镜像创建卷，镜像拷贝一段时间后报错：Failed to copy metadata
to volume: Glance metadata cannot be updated, key signature_verified exists for
volume id b75f957c-0ba4-4099-8706-866c68012779，最终导致创建卷失败。
解决办法：
vim /etc/kolla/cinder-volume/cinder.conf
[DEFAULT]
verify_glance_signatures = disabled
docker restart cinder_volume
-
调整实例大小报危险，发生错误稍后重试
查看horizon日志/var/lib/docker/volumes/kolla_logs/_data/horizon/horizon.log如下：
解决办法：
https://review.opendev.org/c/openstack/horizon/+/829686/ -
虚拟机热迁移same host问题
nova-compute日志报错：
ERROR nova.virt.libvirt.driver [req-acade39d-5762-4ba6-809b-883d55323aff - - - - -] [instance: ae3d9d67-1520-4ebd-b1e3-891a7a53a291] Live Migration failure: internal error: Attempt to migrate guest to the same host 03000200-0400-0500-0006-000700080009: libvirt.libvirtError: internal error: Attempt to migrate guest to the same host 03000200-0400-0500-0006-000700080009
解决办法：
各计算节点通过uuidgen生成UUID
vim /etc/kolla/nova-libvirt/libvirtd.conf
host_uuid = "c536f595-8eb8-4a72-a0da-46ceca198576"
docker restart nova_compute nova_libvirt
8. 参考文献
https://github.com/Huawei/OpenStack_Driver/tree/master/Cinder
https://github.com/Huawei/OpenStack_Driver/tree/50ff55d2cad7d70bb4d26f5975326c2061556c41/ConfigDoc/zh
https://docs.openstack.org/cinder/queens/configuration/block-storage/drivers/huawei-storage-driver.html
https://blog.csdn.net/weixin_34111819/article/details/91730809
鲲鹏展翅 立根铸魂 深耕行业数字化
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)