What is the way to go if you expect the connection to devices in the testbed file to fail?

[JanisFalkenhagenWork]

Hey there,

i am currently developing some tests for my company's infrastructure and because it's a pretty much constantly changing network, failed connects are expected and do not indicate wrong behaviour.

If i then use a testbed which contains all devices, the typical testbed.connect() and further usage of the devices inside of that testbed fails.
See the example below for a quick demo.

[...]
class CommonSetup(aetest.CommonSetup):
    @aetest.subsection
    def connect_to_devs(self, testbed):
        testbed.connect() 
# Here a log message is written to the cli containing the traceback of an internal exception that happens when connection to the device. It 
#is NOT propagated to this context. Otherwise, i could remove the device dynamically from the testbed. It looks like the remaining 
#connections still don't get established. See also Issue #226.

class Testcase(aetest.Testcase):
    @aetest.setup
    def setup(self, testbed):
        self.results = testbed.parse("<sh something>")

# Now when i run this, i get an error:
#3299:  Caught an exception while executing section setup:
#3300:  Traceback (most recent call last):
#3301:    File "TESTSDIR/TEST_SCRIPT_NAME", line LineNo, in setup
#3302:      self.results = testbed.parse(f"<sh something>")
#3303:                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
#3304:    File "src/pyats/topology/testbed.py", line 816, in pyats.topology.testbed.Testbed.parse
#3305:   TypeError: 'NoneType' object is not iterable

    @aetest.test
    def test(self):
        [...]

Is there a way for me to have the testbed.parse command only operate on devices, that have been connected?
Is there a way to force connect to all devices, even though there are some devices that can't be connected to atm?
Is there a way for me to see if a device inside the testbed isn't connected successfully, so i can remove it dynamically?

Is this just bad test design on my part?

I'm looking forward to your answers and thank you for your time. :D

Have a nice day
Janis

[JanisFalkenhagenWork]

Ahh and im on the current version:

You are currently running pyATS version: 24.2
Python: 3.11.5 [64bit]

Package Version

---

genie 24.2
genie.libs.clean 24.2
genie.libs.conf 24.2
genie.libs.filetransferutils 24.2
genie.libs.health 24.2
genie.libs.ops 24.2
genie.libs.parser 24.2
genie.libs.robot 24.2
genie.libs.sdk 24.2
genie.telemetry 24.2
genie.trafficgen 24.2
pyats 24.2
pyats.aereport 24.2
pyats.aetest 24.2
pyats.async 24.2
pyats.connections 24.2
pyats.contrib 24.2
pyats.datastructures 24.2
pyats.easypy 24.2
pyats.kleenex 24.2
pyats.log 24.2
pyats.reporter 24.2
pyats.results 24.2
pyats.robot 24.2
pyats.tcl 24.2
pyats.topology 24.2
pyats.utils 24.2
rest.connector 24.2
unicon 24.2
unicon.plugins 24.2
yang.connector 24.2

[Neeraja1bala]

Hello @JanisFalkenhagenWork ,

Kindly give some time. will check and update you.

[JanisFalkenhagenWork]

Hey,

for now i used this workaround:

class CommonSetup(aetest.CommonSetup)
    @aetest.subsection
    def connect_to_devices(self, testbed):
        dev_to_remove = set()
        futures = []
        with ThreadPoolExecutor() as executor:
            for dev_name in testbed.devices.keys():
                dev = testbed.devices[dev_name]
                futures.append(
                    executor.submit(self._connect_device, dev, dev_name)
                )
        
        for future in futures:
            success, dev_name = future.result()
            if not success:
                dev_to_remove.add(dev_name)
                testbed.remove_device(dev_name)
        
        logging.warning("The following devices couldnt be connected to:")
        logging.warning(dev_to_remove)
        
    def _connect_device(self, dev_obj, dev_name):
        success = True
        try:
            dev_obj.connect()
        except Exception as e:
            logging.warning(f"Device {dev_name} failed to connect. Removing it from the testbed.")
            success = False
        return success, dev_name

It makes the code less concise but seems to work. Also we modifiy the testbed in place, what might have sideeffects for other scripts, as i dont think that they are cloned for each testscript.
It also leaves the user vulnerable to concurrency problems. It would be nice, if functionality like this could be made available in the release version. (If it isn't already and we just missed it xD.)

I am still curious about your solution or recommended/endorsed way of solving this problem.

Have a nice day. :D

[sowmyadn010501]

Hi, thanks for the info. Kindly give me some time to check on this with my team.

[sowmyadn010501]

Hi, the testbed.connect() will connect where the devices able to and fails when the devices are unable to connect. As for the Issue #226 its already fixed. Kindly check

[JanisFalkenhagenWork]

I disagree. The connect does not fail on my machine at least. As i've written above, it prints a log message but does not throw an error. This means that the setup step would not fail, the subsequent tests would thus be executed. However, you are right that the connect stops any further execution. So even if i somehow catch the device that is not connectable, any devices afterwards are not connected, meaning, those tests would unneccessarily fail.

Also i am sorry but this does not really answer my question because i asked on how i would handle this issue, if i expect devices not to be connectable. From what i am understanding from your answer, i would guess that i cannot use the testbed.connect() shortcut. But what would be the standard way of doing the connection then if i wanted to implement it by the book? Or is this case just not covered because it usually points to a critically failing network anyways?

And yeah issue #226 is closed and tbh i don't recall why i mentioned it. I think because it complains about Exceptions being thrown which are not anymore. Maybe the fix mentioned in that issue by @SohanTirpude was to remove the exceptions being propagated to the user, i.e. the testscript, which now makes my life more difficult.

Thanks for the response.

[Neeraja1bala]

Hi,

Kindly try with updated pyats as issue #226 is resolved and in this, @SohanTirpude did not removes the exceptions but only showing exceptions for devices which are unable to connect and for other devices are able to connect. So, kindly try.
And we have a method device.is_connected() which we can use to go through all the devices available in testbed and check whether it is connected or not. If it is not, then please remove it from the testbed and continue the job execution.

[JanisFalkenhagenWork]

Hi,

thanks for your answer. I'll try as soon as i get to it. My timetable is pretty full though atm. So please give me some time to test this out.

The device.is_connected() method is a good hint though. Thank you.

Have a nice rest of your week
Janis

[JanisFalkenhagenWork]

But just out of curiosity?

Why is the device.is_connected() method not listed here?

https://pubhub.devnetcloud.com/media/pyats/docs/topology/concept.html

[Neeraja1bala]

Hello Janis,

Thanks for your update, take your time to try this. Once you did, kindly update the status.

[Neeraja1bala]

Hello Janis,

Kindly let us know the update on this issue.

[JanisFalkenhagenWork]

Hi,

i am currently on vacation. Please give me some more time to test this and let me come back to you then.

Thank you in advance.
Janis

[Neeraja1bala]

Hello ,

Kindly let us know, when you will come back from vacation.

[Neeraja1bala]

Hello @JanisFalkenhagenWork ,

Is any update on this issue, kindly let us know.